Hugging Face's logo

Buckets:
hf-doc-build
/
doc-dev

Files
xet
hf-doc-build/doc-dev / accelerate /pr_4134 /en /_app /immutable /nodes /6.BDA0rt35.js
HuggingFaceDocBuilder's picture
HuggingFaceDocBuilder
download
raw
48.4 kB
import"../chunks/DsnmJJEf.js";import{i as rl,h as pl,C as Ml,H as c,a,E as il,s as hl}from"../chunks/ClLAY5C0.js";import{p as yl,o as Jl,s as l,f as dl,a as t,b as wl,c as o,n as Tl}from"../chunks/BhevU81Y.js";import{T as r}from"../chunks/L24edx8-.js";const Ul='{"title":"Launching distributed training from Jupyter Notebooks","local":"launching-distributed-training-from-jupyter-notebooks","sections":[{"title":"Configuring the Environment","local":"configuring-the-environment","sections":[],"depth":2},{"title":"Preparing the Dataset and Model","local":"preparing-the-dataset-and-model","sections":[],"depth":2},{"title":"Writing the Training Function","local":"writing-the-training-function","sections":[],"depth":2},{"title":"Using the notebook_launcher","local":"using-the-notebooklauncher","sections":[],"depth":2},{"title":"Debugging","local":"debugging","sections":[],"depth":2},{"title":"Conclusion","local":"conclusion","sections":[],"depth":2}],"depth":1}';var ml=o('<meta name="hf:doc:metadata"/>'),jl=o('<p>This tutorial is also available as a Jupyter Notebook <a href="https://github.com/huggingface/notebooks/blob/main/examples/accelerate_examples/simple_cv_example.ipynb" rel="nofollow">here</a></p>'),bl=o("<p>CUDA and XPU can’t be initialized more than once on a multi-device system. It’s fine to debug in the notebook and have calls to CUDA/XPU, but in order to finally train a full cleanup and restart will need to be performed.</p>"),Il=o("<p>The code has been split up to allow for explanations on each section. A full version that can be copy and pasted will be available at the end</p>"),ul=o(`<p>If training on the TPU, your training loop should take in the model as a parameter and it should be instantiated
outside of the training loop function. See the <a href="../concept_guides/training_tpu">TPU best practices</a> to learn why</p>`),Cl=o("<p>You build the model here so that the seed also controls the new weight initialization</p>"),gl=o("<p>There is no specific order to remember, you just need to unpack the objects in the same order you gave them to the prepare method.</p>"),Zl=o("<p>Notice how the <code>node_rank</code> has changed</p>"),fl=o(`<p></p> <!> <!> <p>This tutorial teaches you how to fine-tune a computer vision model with 🤗 Accelerate from a Jupyter Notebook on a distributed system.
You will also learn how to set up a few requirements needed for ensuring your environment is configured properly, your data has been prepared properly, and finally how to launch training.</p> <!> <!> <p>Before any training can be performed, an Accelerate config file must exist in the system. Usually this can be done by running the following in a terminal and answering the prompts:</p> <!> <p>However, if general defaults are fine and you are <em>not</em> running on a TPU, Accelerate has a utility to quickly write your device configuration into a config file via <a href="/docs/accelerate/pr_4134/en/package_reference/utilities#accelerate.commands.config.default.write_basic_config">utils.write_basic_config()</a>.</p> <p>The following code will restart Jupyter after writing the configuration, as CUDA runtime or XPU runtime was called to perform this.</p> <!> <!> <!> <p>Next you should prepare your dataset. As mentioned earlier, great care should be taken when preparing the <code>DataLoaders</code> and model to make sure that <strong>nothing</strong> is put on <em>any</em> GPU.</p> <p>If you do, it is recommended to put that specific code into a function and call that from within the notebook launcher interface, which will be shown later.</p> <p>Make sure the dataset is downloaded based on the directions <a href="https://github.com/huggingface/accelerate/tree/main/examples#simple-vision-example" rel="nofollow">here</a></p> <!> <p>First you need to create a function to extract the class name based on a filename:</p> <!> <!> <p>In the case here, the label is <code>beagle</code>. Using regex you can extract the label from the filename:</p> <!> <!> <p>And you can see it properly returned the right name for our file:</p> <!> <p>Next a <code>Dataset</code> class should be made to handle grabbing the image and the label:</p> <!> <p>Now to build the dataset. Outside the training function you can find and declare all the filenames and labels and use them as references inside the
launched function:</p> <!> <p>Next gather all the labels:</p> <!> <p>Next, you should make a <code>get_dataloaders</code> function that will return your built dataloaders for you. As mentioned earlier, if data is automatically
sent to the GPU or a TPU device when building your <code>DataLoaders</code>, they must be built using this method.</p> <!> <p>Finally, you should import the scheduler to be used later:</p> <!> <!> <p>Now you can build the training loop. <a href="/docs/accelerate/pr_4134/en/package_reference/launchers#accelerate.notebook_launcher">notebook_launcher()</a> works by passing in a function to call that will be ran across the distributed system.</p> <p>Here is a basic training loop for the animal classification problem:</p> <!> <!> <p>First you should set the seed and create an <a href="/docs/accelerate/pr_4134/en/package_reference/accelerator#accelerate.Accelerator">Accelerator</a> object as early in the training loop as possible.</p> <!> <p>Next you should build your dataloaders and create your model:</p> <!> <!> <p>As you are performing transfer learning in this example, the encoder of the model starts out frozen so the head of the model can be
trained only initially:</p> <!> <p>Normalizing the batches of images will make training a little faster:</p> <!> <p>To make these constants available on the active device, you should set it to the Accelerator’s device:</p> <!> <p>Next instantiate the rest of the PyTorch classes used for training:</p> <!> <p>Before passing everything to <a href="/docs/accelerate/pr_4134/en/package_reference/accelerator#accelerate.Accelerator.prepare">prepare()</a>.</p> <!> <!> <p>Now train the model:</p> <!> <p>The evaluation loop will look slightly different compared to the training loop. The number of elements passed as well as the overall
total accuracy of each batch will be added to two constants:</p> <!> <p>Next you have the rest of your standard PyTorch loop:</p> <!> <p>Before finally the last major difference.</p> <p>When performing distributed evaluation, the predictions and labels need to be passed through <a href="/docs/accelerate/pr_4134/en/package_reference/accelerator#accelerate.Accelerator.gather">gather()</a> so that all of the data is available on the current device and a properly calculated metric can be achieved:</p> <!> <p>Now you just need to calculate the actual metric for this problem, and you can print it on the main process using <a href="/docs/accelerate/pr_4134/en/package_reference/accelerator#accelerate.Accelerator.print">print()</a>:</p> <!> <p>A full version of this training loop is available below:</p> <!> <!> <p>All that’s left is to use the <a href="/docs/accelerate/pr_4134/en/package_reference/launchers#accelerate.notebook_launcher">notebook_launcher()</a>.</p> <p>You pass in the function, the arguments (as a tuple), and the number of processes to train on. (See the <a href="../package_reference/launchers">documentation</a> for more information)</p> <!> <!> <p>In the case of running on multiple nodes, you need to set up a Jupyter session at each node and run the launching cell at the same time.</p> <p>For an environment containing 2 nodes (computers) with 8 GPUs each and the main computer with an IP address of “172.31.43.8”, it would look like so:</p> <!> <p>And in the second Jupyter session on the other machine:</p> <!> <!> <p>In the case of running on the TPU, it would look like so:</p> <!> <p>To launch the training process with elasticity, enabling fault tolerance, you can use the <code>elastic_launch</code> feature provided by PyTorch. This requires setting additional parameters such as <code>rdzv_backend</code> and <code>max_restarts</code>. Here is an example of how to use <code>notebook_launcher</code> with elastic capabilities:</p> <!> <p>As it’s running it will print the progress as well as state how many devices you ran on. This tutorial was ran with two GPUs:</p> <!> <p>And that’s it!</p> <p>Please note that <a href="/docs/accelerate/pr_4134/en/package_reference/launchers#accelerate.notebook_launcher">notebook_launcher()</a> ignores the Accelerate config file, to launch based on the config use:</p> <!> <!> <p>A common issue when running the <code>notebook_launcher</code> is receiving a CUDA/XPU has already been initialized issue. This usually stems
from an import or prior code in the notebook that makes a call to the PyTorch <code>torch.cuda</code> or <code>torch.xpu</code> sublibrary. To help narrow down what went wrong,
you can launch the <code>notebook_launcher</code> with <code>ACCELERATE_DEBUG_MODE=yes</code> in your environment and an additional check
will be made when spawning that a regular process can be created and utilize CUDA/XPU without issue. (Your CUDA/XPU code can still be ran afterwards).</p> <!> <p>This notebook showed how to perform distributed training from inside of a Jupyter Notebook. Some key notes to remember:</p> <ul><li>Make sure to save any code that use CUDA/XPU (or CUDA/XPU imports) for the function passed to <a href="/docs/accelerate/pr_4134/en/package_reference/launchers#accelerate.notebook_launcher">notebook_launcher()</a></li> <li>Set the <code>num_processes</code> to be the number of devices used for training (such as number of GPUs, XPUs, CPUs, TPUs, etc)</li> <li>If using the TPU, declare your model outside the training loop function</li></ul> <!> <p></p>`,1);function kl(tl,ol){yl(ol,!1),Jl(()=>{new URLSearchParams(window.location.search).get("fw")}),rl();var p=fl();pl("nxncwf",s=>{var n=ml();hl(n,"content",Ul),t(s,n)});var M=l(dl(p),2);Ml(M,{containerStyle:"float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"});var i=l(M,2);c(i,{title:"Launching distributed training from Jupyter Notebooks",local:"launching-distributed-training-from-jupyter-notebooks",headingTag:"h1"});var h=l(i,4);r(h,{children:(s,n)=>{var e=jl();t(s,e)},$$slots:{default:!0}});var y=l(h,2);c(y,{title:"Configuring the Environment",local:"configuring-the-environment",headingTag:"h2"});var J=l(y,4);a(J,{code:"YWNjZWxlcmF0ZSUyMGNvbmZpZw==",highlighted:"accelerate config",lang:"bash",wrap:!1});var d=l(J,6);r(d,{warning:!0,children:(s,n)=>{var e=bl();t(s,e)},$$slots:{default:!0}});var w=l(d,2);a(w,{code:"aW1wb3J0JTIwb3MlMEFmcm9tJTIwYWNjZWxlcmF0ZS51dGlscyUyMGltcG9ydCUyMHdyaXRlX2Jhc2ljX2NvbmZpZyUwQSUwQXdyaXRlX2Jhc2ljX2NvbmZpZygpJTIwJTIwJTIzJTIwV3JpdGUlMjBhJTIwY29uZmlnJTIwZmlsZSUwQW9zLl9leGl0KDAwKSUyMCUyMCUyMyUyMFJlc3RhcnQlMjB0aGUlMjBub3RlYm9vaw==",highlighted:`<span class="hljs-keyword">import</span> os
<span class="hljs-keyword">from</span> accelerate.utils <span class="hljs-keyword">import</span> write_basic_config
write_basic_config() <span class="hljs-comment"># Write a config file</span>
os._exit(<span class="hljs-number">00</span>) <span class="hljs-comment"># Restart the notebook</span>`,lang:"python",wrap:!1});var T=l(w,2);c(T,{title:"Preparing the Dataset and Model",local:"preparing-the-dataset-and-model",headingTag:"h2"});var U=l(T,8);a(U,{code:"aW1wb3J0JTIwb3MlMkMlMjByZSUyQyUyMHRvcmNoJTJDJTIwUElMJTBBaW1wb3J0JTIwbnVtcHklMjBhcyUyMG5wJTBBJTBBZnJvbSUyMHRvcmNoLm9wdGltLmxyX3NjaGVkdWxlciUyMGltcG9ydCUyME9uZUN5Y2xlTFIlMEFmcm9tJTIwdG9yY2gudXRpbHMuZGF0YSUyMGltcG9ydCUyMERhdGFMb2FkZXIlMkMlMjBEYXRhc2V0JTBBZnJvbSUyMHRvcmNodmlzaW9uLnRyYW5zZm9ybXMlMjBpbXBvcnQlMjBDb21wb3NlJTJDJTIwUmFuZG9tUmVzaXplZENyb3AlMkMlMjBSZXNpemUlMkMlMjBUb1RlbnNvciUwQSUwQWZyb20lMjBhY2NlbGVyYXRlJTIwaW1wb3J0JTIwQWNjZWxlcmF0b3IlMEFmcm9tJTIwYWNjZWxlcmF0ZS51dGlscyUyMGltcG9ydCUyMHNldF9zZWVkJTBBZnJvbSUyMHRpbW0lMjBpbXBvcnQlMjBjcmVhdGVfbW9kZWw=",highlighted:`<span class="hljs-keyword">import</span> os, re, torch, PIL
<span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">from</span> torch.optim.lr_scheduler <span class="hljs-keyword">import</span> OneCycleLR
<span class="hljs-keyword">from</span> torch.utils.data <span class="hljs-keyword">import</span> DataLoader, Dataset
<span class="hljs-keyword">from</span> torchvision.transforms <span class="hljs-keyword">import</span> Compose, RandomResizedCrop, Resize, ToTensor
<span class="hljs-keyword">from</span> accelerate <span class="hljs-keyword">import</span> Accelerator
<span class="hljs-keyword">from</span> accelerate.utils <span class="hljs-keyword">import</span> set_seed
<span class="hljs-keyword">from</span> timm <span class="hljs-keyword">import</span> create_model`,lang:"python",wrap:!1});var m=l(U,4);a(m,{code:"aW1wb3J0JTIwb3MlMEElMEFkYXRhX2RpciUyMCUzRCUyMCUyMi4uJTJGLi4lMkZpbWFnZXMlMjIlMEFmbmFtZXMlMjAlM0QlMjBvcy5saXN0ZGlyKGRhdGFfZGlyKSUwQWZuYW1lJTIwJTNEJTIwZm5hbWVzJTVCMCU1RCUwQXByaW50KGZuYW1lKQ==",highlighted:`<span class="hljs-keyword">import</span> os
data_dir = <span class="hljs-string">&quot;../../images&quot;</span>
fnames = os.listdir(data_dir)
fname = fnames[<span class="hljs-number">0</span>]
<span class="hljs-built_in">print</span>(fname)`,lang:"python",wrap:!1});var j=l(m,2);a(j,{code:"YmVhZ2xlXzMyLmpwZw==",highlighted:"beagle_32.jpg",lang:"python",wrap:!1});var b=l(j,4);a(b,{code:"aW1wb3J0JTIwcmUlMEElMEElMEFkZWYlMjBleHRyYWN0X2xhYmVsKGZuYW1lKSUzQSUwQSUyMCUyMCUyMCUyMHN0ZW0lMjAlM0QlMjBmbmFtZS5zcGxpdChvcy5wYXRoLnNlcCklNUItMSU1RCUwQSUyMCUyMCUyMCUyMHJldHVybiUyMHJlLnNlYXJjaChyJTIyJTVFKC4qKV8lNUNkJTJCJTVDLmpwZyUyNCUyMiUyQyUyMHN0ZW0pLmdyb3VwcygpJTVCMCU1RA==",highlighted:`<span class="hljs-keyword">import</span> re
<span class="hljs-keyword">def</span> <span class="hljs-title function_">extract_label</span>(<span class="hljs-params">fname</span>):
 stem = fname.split(os.path.sep)[-<span class="hljs-number">1</span>]
 <span class="hljs-keyword">return</span> re.search(<span class="hljs-string">r&quot;^(.*)_\\d+\\.jpg$&quot;</span>, stem).groups()[<span class="hljs-number">0</span>]`,lang:"python",wrap:!1});var I=l(b,2);a(I,{code:"ZXh0cmFjdF9sYWJlbChmbmFtZSk=",highlighted:"extract_label(fname)",lang:"python",wrap:!1});var u=l(I,4);a(u,{code:"JTIyYmVhZ2xlJTIy",highlighted:'<span class="hljs-string">&quot;beagle&quot;</span>',lang:"python",wrap:!1});var C=l(u,4);a(C,{code:"Y2xhc3MlMjBQZXRzRGF0YXNldChEYXRhc2V0KSUzQSUwQSUyMCUyMCUyMCUyMGRlZiUyMF9faW5pdF9fKHNlbGYlMkMlMjBmaWxlX25hbWVzJTJDJTIwaW1hZ2VfdHJhbnNmb3JtJTNETm9uZSUyQyUyMGxhYmVsX3RvX2lkJTNETm9uZSklM0ElMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBzZWxmLmZpbGVfbmFtZXMlMjAlM0QlMjBmaWxlX25hbWVzJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwc2VsZi5pbWFnZV90cmFuc2Zvcm0lMjAlM0QlMjBpbWFnZV90cmFuc2Zvcm0lMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBzZWxmLmxhYmVsX3RvX2lkJTIwJTNEJTIwbGFiZWxfdG9faWQlMEElMEElMjAlMjAlMjAlMjBkZWYlMjBfX2xlbl9fKHNlbGYpJTNBJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwcmV0dXJuJTIwbGVuKHNlbGYuZmlsZV9uYW1lcyklMEElMEElMjAlMjAlMjAlMjBkZWYlMjBfX2dldGl0ZW1fXyhzZWxmJTJDJTIwaWR4KSUzQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGZuYW1lJTIwJTNEJTIwc2VsZi5maWxlX25hbWVzJTVCaWR4JTVEJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwcmF3X2ltYWdlJTIwJTNEJTIwUElMLkltYWdlLm9wZW4oZm5hbWUpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwaW1hZ2UlMjAlM0QlMjByYXdfaW1hZ2UuY29udmVydCglMjJSR0IlMjIpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwaWYlMjBzZWxmLmltYWdlX3RyYW5zZm9ybSUyMGlzJTIwbm90JTIwTm9uZSUzQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGltYWdlJTIwJTNEJTIwc2VsZi5pbWFnZV90cmFuc2Zvcm0oaW1hZ2UpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbGFiZWwlMjAlM0QlMjBleHRyYWN0X2xhYmVsKGZuYW1lKSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGlmJTIwc2VsZi5sYWJlbF90b19pZCUyMGlzJTIwbm90JTIwTm9uZSUzQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGxhYmVsJTIwJTNEJTIwc2VsZi5sYWJlbF90b19pZCU1QmxhYmVsJTVEJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwcmV0dXJuJTIwJTdCJTIyaW1hZ2UlMjIlM0ElMjBpbWFnZSUyQyUyMCUyMmxhYmVsJTIyJTNBJTIwbGFiZWwlN0Q=",highlighted:`<span class="hljs-keyword">class</span> <span class="hljs-title class_">PetsDataset</span>(<span class="hljs-title class_ inherited__">Dataset</span>):
 <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, file_names, image_transform=<span class="hljs-literal">None</span>, label_to_id=<span class="hljs-literal">None</span></span>):
 <span class="hljs-variable language_">self</span>.file_names = file_names
 <span class="hljs-variable language_">self</span>.image_transform = image_transform
 <span class="hljs-variable language_">self</span>.label_to_id = label_to_id
 <span class="hljs-keyword">def</span> <span class="hljs-title function_">__len__</span>(<span class="hljs-params">self</span>):
 <span class="hljs-keyword">return</span> <span class="hljs-built_in">len</span>(<span class="hljs-variable language_">self</span>.file_names)
 <span class="hljs-keyword">def</span> <span class="hljs-title function_">__getitem__</span>(<span class="hljs-params">self, idx</span>):
 fname = <span class="hljs-variable language_">self</span>.file_names[idx]
 raw_image = PIL.Image.<span class="hljs-built_in">open</span>(fname)
 image = raw_image.convert(<span class="hljs-string">&quot;RGB&quot;</span>)
 <span class="hljs-keyword">if</span> <span class="hljs-variable language_">self</span>.image_transform <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:
 image = <span class="hljs-variable language_">self</span>.image_transform(image)
 label = extract_label(fname)
 <span class="hljs-keyword">if</span> <span class="hljs-variable language_">self</span>.label_to_id <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:
 label = <span class="hljs-variable language_">self</span>.label_to_id[label]
 <span class="hljs-keyword">return</span> {<span class="hljs-string">&quot;image&quot;</span>: image, <span class="hljs-string">&quot;label&quot;</span>: label}`,lang:"python",wrap:!1});var g=l(C,4);a(g,{code:"Zm5hbWVzJTIwJTNEJTIwJTVCb3MucGF0aC5qb2luKCUyMi4uJTJGLi4lMkZpbWFnZXMlMjIlMkMlMjBmbmFtZSklMjBmb3IlMjBmbmFtZSUyMGluJTIwZm5hbWVzJTIwaWYlMjBmbmFtZS5lbmRzd2l0aCglMjIuanBnJTIyKSU1RA==",highlighted:'fnames = [os.path.join(<span class="hljs-string">&quot;../../images&quot;</span>, fname) <span class="hljs-keyword">for</span> fname <span class="hljs-keyword">in</span> fnames <span class="hljs-keyword">if</span> fname.endswith(<span class="hljs-string">&quot;.jpg&quot;</span>)]',lang:"python",wrap:!1});var Z=l(g,4);a(Z,{code:"YWxsX2xhYmVscyUyMCUzRCUyMCU1QmV4dHJhY3RfbGFiZWwoZm5hbWUpJTIwZm9yJTIwZm5hbWUlMjBpbiUyMGZuYW1lcyU1RCUwQWlkX3RvX2xhYmVsJTIwJTNEJTIwbGlzdChzZXQoYWxsX2xhYmVscykpJTBBaWRfdG9fbGFiZWwuc29ydCgpJTBBbGFiZWxfdG9faWQlMjAlM0QlMjAlN0JsYmwlM0ElMjBpJTIwZm9yJTIwaSUyQyUyMGxibCUyMGluJTIwZW51bWVyYXRlKGlkX3RvX2xhYmVsKSU3RA==",highlighted:`all_labels = [extract_label(fname) <span class="hljs-keyword">for</span> fname <span class="hljs-keyword">in</span> fnames]
id_to_label = <span class="hljs-built_in">list</span>(<span class="hljs-built_in">set</span>(all_labels))
id_to_label.sort()
label_to_id = {lbl: i <span class="hljs-keyword">for</span> i, lbl <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(id_to_label)}`,lang:"python",wrap:!1});var f=l(Z,4);a(f,{code:"ZGVmJTIwZ2V0X2RhdGFsb2FkZXJzKGJhdGNoX3NpemUlM0ElMjBpbnQlMjAlM0QlMjA2NCklM0ElMEElMjAlMjAlMjAlMjAlMjJCdWlsZHMlMjBhJTIwc2V0JTIwb2YlMjBkYXRhbG9hZGVycyUyMHdpdGglMjBhJTIwYmF0Y2hfc2l6ZSUyMiUwQSUyMCUyMCUyMCUyMHJhbmRvbV9wZXJtJTIwJTNEJTIwbnAucmFuZG9tLnBlcm11dGF0aW9uKGxlbihmbmFtZXMpKSUwQSUyMCUyMCUyMCUyMGN1dCUyMCUzRCUyMGludCgwLjglMjAqJTIwbGVuKGZuYW1lcykpJTBBJTIwJTIwJTIwJTIwdHJhaW5fc3BsaXQlMjAlM0QlMjByYW5kb21fcGVybSU1QiUzQWN1dCU1RCUwQSUyMCUyMCUyMCUyMGV2YWxfc3BsaXQlMjAlM0QlMjByYW5kb21fcGVybSU1QmN1dCUzQSU1RCUwQSUwQSUyMCUyMCUyMCUyMCUyMyUyMEZvciUyMHRyYWluaW5nJTIwYSUyMHNpbXBsZSUyMFJhbmRvbVJlc2l6ZWRDcm9wJTIwd2lsbCUyMGJlJTIwdXNlZCUwQSUyMCUyMCUyMCUyMHRyYWluX3RmbSUyMCUzRCUyMENvbXBvc2UoJTVCUmFuZG9tUmVzaXplZENyb3AoKDIyNCUyQyUyMDIyNCklMkMlMjBzY2FsZSUzRCgwLjUlMkMlMjAxLjApKSUyQyUyMFRvVGVuc29yKCklNUQpJTBBJTIwJTIwJTIwJTIwdHJhaW5fZGF0YXNldCUyMCUzRCUyMFBldHNEYXRhc2V0KCU1QmZuYW1lcyU1QmklNUQlMjBmb3IlMjBpJTIwaW4lMjB0cmFpbl9zcGxpdCU1RCUyQyUyMGltYWdlX3RyYW5zZm9ybSUzRHRyYWluX3RmbSUyQyUyMGxhYmVsX3RvX2lkJTNEbGFiZWxfdG9faWQpJTBBJTBBJTIwJTIwJTIwJTIwJTIzJTIwRm9yJTIwZXZhbHVhdGlvbiUyMGElMjBkZXRlcm1pbmlzdGljJTIwUmVzaXplJTIwd2lsbCUyMGJlJTIwdXNlZCUwQSUyMCUyMCUyMCUyMGV2YWxfdGZtJTIwJTNEJTIwQ29tcG9zZSglNUJSZXNpemUoKDIyNCUyQyUyMDIyNCkpJTJDJTIwVG9UZW5zb3IoKSU1RCklMEElMjAlMjAlMjAlMjBldmFsX2RhdGFzZXQlMjAlM0QlMjBQZXRzRGF0YXNldCglNUJmbmFtZXMlNUJpJTVEJTIwZm9yJTIwaSUyMGluJTIwZXZhbF9zcGxpdCU1RCUyQyUyMGltYWdlX3RyYW5zZm9ybSUzRGV2YWxfdGZtJTJDJTIwbGFiZWxfdG9faWQlM0RsYWJlbF90b19pZCklMEElMEElMjAlMjAlMjAlMjAlMjMlMjBJbnN0YW50aWF0ZSUyMGRhdGFsb2FkZXJzJTBBJTIwJTIwJTIwJTIwdHJhaW5fZGF0YWxvYWRlciUyMCUzRCUyMERhdGFMb2FkZXIodHJhaW5fZGF0YXNldCUyQyUyMHNodWZmbGUlM0RUcnVlJTJDJTIwYmF0Y2hfc2l6ZSUzRGJhdGNoX3NpemUlMkMlMjBudW1fd29ya2VycyUzRDQpJTBBJTIwJTIwJTIwJTIwZXZhbF9kYXRhbG9hZGVyJTIwJTNEJTIwRGF0YUxvYWRlcihldmFsX2RhdGFzZXQlMkMlMjBzaHVmZmxlJTNERmFsc2UlMkMlMjBiYXRjaF9zaXplJTNEYmF0Y2hfc2l6ZSUyMColMjAyJTJDJTIwbnVtX3dvcmtlcnMlM0Q0KSUwQSUyMCUyMCUyMCUyMHJldHVybiUyMHRyYWluX2RhdGFsb2FkZXIlMkMlMjBldmFsX2RhdGFsb2FkZXI=",highlighted:`<span class="hljs-keyword">def</span> <span class="hljs-title function_">get_dataloaders</span>(<span class="hljs-params">batch_size: <span class="hljs-built_in">int</span> = <span class="hljs-number">64</span></span>):
 <span class="hljs-string">&quot;Builds a set of dataloaders with a batch_size&quot;</span>
 random_perm = np.random.permutation(<span class="hljs-built_in">len</span>(fnames))
 cut = <span class="hljs-built_in">int</span>(<span class="hljs-number">0.8</span> * <span class="hljs-built_in">len</span>(fnames))
 train_split = random_perm[:cut]
 eval_split = random_perm[cut:]
 <span class="hljs-comment"># For training a simple RandomResizedCrop will be used</span>
 train_tfm = Compose([RandomResizedCrop((<span class="hljs-number">224</span>, <span class="hljs-number">224</span>), scale=(<span class="hljs-number">0.5</span>, <span class="hljs-number">1.0</span>)), ToTensor()])
 train_dataset = PetsDataset([fnames[i] <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> train_split], image_transform=train_tfm, label_to_id=label_to_id)
 <span class="hljs-comment"># For evaluation a deterministic Resize will be used</span>
 eval_tfm = Compose([Resize((<span class="hljs-number">224</span>, <span class="hljs-number">224</span>)), ToTensor()])
 eval_dataset = PetsDataset([fnames[i] <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> eval_split], image_transform=eval_tfm, label_to_id=label_to_id)
 <span class="hljs-comment"># Instantiate dataloaders</span>
 train_dataloader = DataLoader(train_dataset, shuffle=<span class="hljs-literal">True</span>, batch_size=batch_size, num_workers=<span class="hljs-number">4</span>)
 eval_dataloader = DataLoader(eval_dataset, shuffle=<span class="hljs-literal">False</span>, batch_size=batch_size * <span class="hljs-number">2</span>, num_workers=<span class="hljs-number">4</span>)
 <span class="hljs-keyword">return</span> train_dataloader, eval_dataloader`,lang:"python",wrap:!1});var G=l(f,4);a(G,{code:"ZnJvbSUyMHRvcmNoLm9wdGltLmxyX3NjaGVkdWxlciUyMGltcG9ydCUyMENvc2luZUFubmVhbGluZ0xS",highlighted:'<span class="hljs-keyword">from</span> torch.optim.lr_scheduler <span class="hljs-keyword">import</span> CosineAnnealingLR',lang:"python",wrap:!1});var _=l(G,2);c(_,{title:"Writing the Training Function",local:"writing-the-training-function",headingTag:"h2"});var W=l(_,6);r(W,{children:(s,n)=>{var e=Il();t(s,e)},$$slots:{default:!0}});var B=l(W,2);a(B,{code:"ZGVmJTIwdHJhaW5pbmdfbG9vcChtaXhlZF9wcmVjaXNpb24lM0QlMjJmcDE2JTIyJTJDJTIwc2VlZCUzQSUyMGludCUyMCUzRCUyMDQyJTJDJTIwYmF0Y2hfc2l6ZSUzQSUyMGludCUyMCUzRCUyMDY0KSUzQSUwQSUyMCUyMCUyMCUyMHNldF9zZWVkKHNlZWQpJTBBJTIwJTIwJTIwJTIwYWNjZWxlcmF0b3IlMjAlM0QlMjBBY2NlbGVyYXRvcihtaXhlZF9wcmVjaXNpb24lM0RtaXhlZF9wcmVjaXNpb24p",highlighted:`<span class="hljs-keyword">def</span> <span class="hljs-title function_">training_loop</span>(<span class="hljs-params">mixed_precision=<span class="hljs-string">&quot;fp16&quot;</span>, seed: <span class="hljs-built_in">int</span> = <span class="hljs-number">42</span>, batch_size: <span class="hljs-built_in">int</span> = <span class="hljs-number">64</span></span>):
 set_seed(seed)
 accelerator = Accelerator(mixed_precision=mixed_precision)`,lang:"python",wrap:!1});var k=l(B,4);r(k,{warning:!0,children:(s,n)=>{var e=ul();t(s,e)},$$slots:{default:!0}});var X=l(k,4);a(X,{code:"JTIwJTIwJTIwJTIwdHJhaW5fZGF0YWxvYWRlciUyQyUyMGV2YWxfZGF0YWxvYWRlciUyMCUzRCUyMGdldF9kYXRhbG9hZGVycyhiYXRjaF9zaXplKSUwQSUyMCUyMCUyMCUyMG1vZGVsJTIwJTNEJTIwY3JlYXRlX21vZGVsKCUyMnJlc25ldDUwZCUyMiUyQyUyMHByZXRyYWluZWQlM0RUcnVlJTJDJTIwbnVtX2NsYXNzZXMlM0RsZW4obGFiZWxfdG9faWQpKQ==",highlighted:` train_dataloader, eval_dataloader = get_dataloaders(batch_size)
 model = create_model(<span class="hljs-string">&quot;resnet50d&quot;</span>, pretrained=<span class="hljs-literal">True</span>, num_classes=<span class="hljs-built_in">len</span>(label_to_id))`,lang:"python",wrap:!1});var Y=l(X,2);r(Y,{children:(s,n)=>{var e=Cl();t(s,e)},$$slots:{default:!0}});var N=l(Y,4);a(N,{code:"JTIwJTIwJTIwJTIwZm9yJTIwcGFyYW0lMjBpbiUyMG1vZGVsLnBhcmFtZXRlcnMoKSUzQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMHBhcmFtLnJlcXVpcmVzX2dyYWQlMjAlM0QlMjBGYWxzZSUwQSUyMCUyMCUyMCUyMGZvciUyMHBhcmFtJTIwaW4lMjBtb2RlbC5nZXRfY2xhc3NpZmllcigpLnBhcmFtZXRlcnMoKSUzQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMHBhcmFtLnJlcXVpcmVzX2dyYWQlMjAlM0QlMjBUcnVl",highlighted:` <span class="hljs-keyword">for</span> param <span class="hljs-keyword">in</span> model.parameters():
 param.requires_grad = <span class="hljs-literal">False</span>
 <span class="hljs-keyword">for</span> param <span class="hljs-keyword">in</span> model.get_classifier().parameters():
 param.requires_grad = <span class="hljs-literal">True</span>`,lang:"python",wrap:!1});var A=l(N,4);a(A,{code:"JTIwJTIwJTIwJTIwbWVhbiUyMCUzRCUyMHRvcmNoLnRlbnNvcihtb2RlbC5kZWZhdWx0X2NmZyU1QiUyMm1lYW4lMjIlNUQpJTVCTm9uZSUyQyUyMCUzQSUyQyUyME5vbmUlMkMlMjBOb25lJTVEJTBBJTIwJTIwJTIwJTIwc3RkJTIwJTNEJTIwdG9yY2gudGVuc29yKG1vZGVsLmRlZmF1bHRfY2ZnJTVCJTIyc3RkJTIyJTVEKSU1Qk5vbmUlMkMlMjAlM0ElMkMlMjBOb25lJTJDJTIwTm9uZSU1RA==",highlighted:` mean = torch.tensor(model.default_cfg[<span class="hljs-string">&quot;mean&quot;</span>])[<span class="hljs-literal">None</span>, :, <span class="hljs-literal">None</span>, <span class="hljs-literal">None</span>]
 std = torch.tensor(model.default_cfg[<span class="hljs-string">&quot;std&quot;</span>])[<span class="hljs-literal">None</span>, :, <span class="hljs-literal">None</span>, <span class="hljs-literal">None</span>]`,lang:"python",wrap:!1});var R=l(A,4);a(R,{code:"JTIwJTIwJTIwJTIwbWVhbiUyMCUzRCUyMG1lYW4udG8oYWNjZWxlcmF0b3IuZGV2aWNlKSUwQSUyMCUyMCUyMCUyMHN0ZCUyMCUzRCUyMHN0ZC50byhhY2NlbGVyYXRvci5kZXZpY2Up",highlighted:` mean = mean.to(accelerator.device)
 std = std.to(accelerator.device)`,lang:"python",wrap:!1});var v=l(R,4);a(v,{code:"JTIwJTIwJTIwJTIwb3B0aW1pemVyJTIwJTNEJTIwdG9yY2gub3B0aW0uQWRhbShwYXJhbXMlM0Rtb2RlbC5wYXJhbWV0ZXJzKCklMkMlMjBsciUzRDNlLTIlMjAlMkYlMjAyNSklMEElMjAlMjAlMjAlMjBscl9zY2hlZHVsZXIlMjAlM0QlMjBPbmVDeWNsZUxSKG9wdGltaXplciUzRG9wdGltaXplciUyQyUyMG1heF9sciUzRDNlLTIlMkMlMjBlcG9jaHMlM0Q1JTJDJTIwc3RlcHNfcGVyX2Vwb2NoJTNEbGVuKHRyYWluX2RhdGFsb2FkZXIpKQ==",highlighted:` optimizer = torch.optim.Adam(params=model.parameters(), lr=<span class="hljs-number">3e-2</span> / <span class="hljs-number">25</span>)
 lr_scheduler = OneCycleLR(optimizer=optimizer, max_lr=<span class="hljs-number">3e-2</span>, epochs=<span class="hljs-number">5</span>, steps_per_epoch=<span class="hljs-built_in">len</span>(train_dataloader))`,lang:"python",wrap:!1});var V=l(v,4);r(V,{children:(s,n)=>{var e=gl();t(s,e)},$$slots:{default:!0}});var F=l(V,2);a(F,{code:"JTIwJTIwJTIwJTIwbW9kZWwlMkMlMjBvcHRpbWl6ZXIlMkMlMjB0cmFpbl9kYXRhbG9hZGVyJTJDJTIwZXZhbF9kYXRhbG9hZGVyJTJDJTIwbHJfc2NoZWR1bGVyJTIwJTNEJTIwYWNjZWxlcmF0b3IucHJlcGFyZSglMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBtb2RlbCUyQyUyMG9wdGltaXplciUyQyUyMHRyYWluX2RhdGFsb2FkZXIlMkMlMjBldmFsX2RhdGFsb2FkZXIlMkMlMjBscl9zY2hlZHVsZXIlMEElMjAlMjAlMjAlMjAp",highlighted:` model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
 model, optimizer, train_dataloader, eval_dataloader, lr_scheduler
 )`,lang:"python",wrap:!1});var Q=l(F,4);a(Q,{code:"JTIwJTIwJTIwJTIwZm9yJTIwZXBvY2glMjBpbiUyMHJhbmdlKDUpJTNBJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbW9kZWwudHJhaW4oKSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGZvciUyMGJhdGNoJTIwaW4lMjB0cmFpbl9kYXRhbG9hZGVyJTNBJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwaW5wdXRzJTIwJTNEJTIwKGJhdGNoJTVCJTIyaW1hZ2UlMjIlNUQlMjAtJTIwbWVhbiklMjAlMkYlMjBzdGQlMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBvdXRwdXRzJTIwJTNEJTIwbW9kZWwoaW5wdXRzKSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGxvc3MlMjAlM0QlMjB0b3JjaC5ubi5mdW5jdGlvbmFsLmNyb3NzX2VudHJvcHkob3V0cHV0cyUyQyUyMGJhdGNoJTVCJTIybGFiZWwlMjIlNUQpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwYWNjZWxlcmF0b3IuYmFja3dhcmQobG9zcyklMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBvcHRpbWl6ZXIuc3RlcCgpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbHJfc2NoZWR1bGVyLnN0ZXAoKSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMG9wdGltaXplci56ZXJvX2dyYWQoKQ==",highlighted:` <span class="hljs-keyword">for</span> epoch <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">5</span>):
 model.train()
 <span class="hljs-keyword">for</span> batch <span class="hljs-keyword">in</span> train_dataloader:
 inputs = (batch[<span class="hljs-string">&quot;image&quot;</span>] - mean) / std
 outputs = model(inputs)
 loss = torch.nn.functional.cross_entropy(outputs, batch[<span class="hljs-string">&quot;label&quot;</span>])
 accelerator.backward(loss)
 optimizer.step()
 lr_scheduler.step()
 optimizer.zero_grad()`,lang:"python",wrap:!1});var z=l(Q,4);a(z,{code:"JTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbW9kZWwuZXZhbCgpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwYWNjdXJhdGUlMjAlM0QlMjAwJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbnVtX2VsZW1zJTIwJTNEJTIwMA==",highlighted:` model.<span class="hljs-built_in">eval</span>()
 accurate = <span class="hljs-number">0</span>
 num_elems = <span class="hljs-number">0</span>`,lang:"python",wrap:!1});var E=l(z,4);a(E,{code:"JTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwZm9yJTIwYmF0Y2glMjBpbiUyMGV2YWxfZGF0YWxvYWRlciUzQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGlucHV0cyUyMCUzRCUyMChiYXRjaCU1QiUyMmltYWdlJTIyJTVEJTIwLSUyMG1lYW4pJTIwJTJGJTIwc3RkJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwd2l0aCUyMHRvcmNoLm5vX2dyYWQoKSUzQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMG91dHB1dHMlMjAlM0QlMjBtb2RlbChpbnB1dHMpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwcHJlZGljdGlvbnMlMjAlM0QlMjBvdXRwdXRzLmFyZ21heChkaW0lM0QtMSk=",highlighted:` <span class="hljs-keyword">for</span> batch <span class="hljs-keyword">in</span> eval_dataloader:
 inputs = (batch[<span class="hljs-string">&quot;image&quot;</span>] - mean) / std
 <span class="hljs-keyword">with</span> torch.no_grad():
 outputs = model(inputs)
 predictions = outputs.argmax(dim=-<span class="hljs-number">1</span>)`,lang:"python",wrap:!1});var S=l(E,6);a(S,{code:"JTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwYWNjdXJhdGVfcHJlZHMlMjAlM0QlMjBhY2NlbGVyYXRvci5nYXRoZXIocHJlZGljdGlvbnMpJTIwJTNEJTNEJTIwYWNjZWxlcmF0b3IuZ2F0aGVyKGJhdGNoJTVCJTIybGFiZWwlMjIlNUQpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbnVtX2VsZW1zJTIwJTJCJTNEJTIwYWNjdXJhdGVfcHJlZHMuc2hhcGUlNUIwJTVEJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwYWNjdXJhdGUlMjAlMkIlM0QlMjBhY2N1cmF0ZV9wcmVkcy5sb25nKCkuc3VtKCk=",highlighted:` accurate_preds = accelerator.gather(predictions) == accelerator.gather(batch[<span class="hljs-string">&quot;label&quot;</span>])
 num_elems += accurate_preds.shape[<span class="hljs-number">0</span>]
 accurate += accurate_preds.long().<span class="hljs-built_in">sum</span>()`,lang:"python",wrap:!1});var H=l(S,4);a(H,{code:"JTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwZXZhbF9tZXRyaWMlMjAlM0QlMjBhY2N1cmF0ZS5pdGVtKCklMjAlMkYlMjBudW1fZWxlbXMlMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBhY2NlbGVyYXRvci5wcmludChmJTIyZXBvY2glMjAlN0JlcG9jaCU3RCUzQSUyMCU3QjEwMCUyMColMjBldmFsX21ldHJpYyUzQS4yZiU3RCUyMik=",highlighted:` eval_metric = accurate.item() / num_elems
 accelerator.<span class="hljs-built_in">print</span>(<span class="hljs-string">f&quot;epoch <span class="hljs-subst">{epoch}</span>: <span class="hljs-subst">{<span class="hljs-number">100</span> * eval_metric:<span class="hljs-number">.2</span>f}</span>&quot;</span>)`,lang:"python",wrap:!1});var x=l(H,4);a(x,{code:"ZGVmJTIwdHJhaW5pbmdfbG9vcChtaXhlZF9wcmVjaXNpb24lM0QlMjJmcDE2JTIyJTJDJTIwc2VlZCUzQSUyMGludCUyMCUzRCUyMDQyJTJDJTIwYmF0Y2hfc2l6ZSUzQSUyMGludCUyMCUzRCUyMDY0KSUzQSUwQSUyMCUyMCUyMCUyMHNldF9zZWVkKHNlZWQpJTBBJTIwJTIwJTIwJTIwJTIzJTIwSW5pdGlhbGl6ZSUyMGFjY2VsZXJhdG9yJTBBJTIwJTIwJTIwJTIwYWNjZWxlcmF0b3IlMjAlM0QlMjBBY2NlbGVyYXRvcihtaXhlZF9wcmVjaXNpb24lM0RtaXhlZF9wcmVjaXNpb24pJTBBJTIwJTIwJTIwJTIwJTIzJTIwQnVpbGQlMjBkYXRhbG9hZGVycyUwQSUyMCUyMCUyMCUyMHRyYWluX2RhdGFsb2FkZXIlMkMlMjBldmFsX2RhdGFsb2FkZXIlMjAlM0QlMjBnZXRfZGF0YWxvYWRlcnMoYmF0Y2hfc2l6ZSklMEElMEElMjAlMjAlMjAlMjAlMjMlMjBJbnN0YW50aWF0ZSUyMHRoZSUyMG1vZGVsJTIwKHlvdSUyMGJ1aWxkJTIwdGhlJTIwbW9kZWwlMjBoZXJlJTIwc28lMjB0aGF0JTIwdGhlJTIwc2VlZCUyMGFsc28lMjBjb250cm9scyUyMG5ldyUyMHdlaWdodCUyMGluaXRpYWxpemF0aW9ucyklMEElMjAlMjAlMjAlMjBtb2RlbCUyMCUzRCUyMGNyZWF0ZV9tb2RlbCglMjJyZXNuZXQ1MGQlMjIlMkMlMjBwcmV0cmFpbmVkJTNEVHJ1ZSUyQyUyMG51bV9jbGFzc2VzJTNEbGVuKGxhYmVsX3RvX2lkKSklMEElMEElMjAlMjAlMjAlMjAlMjMlMjBGcmVlemUlMjB0aGUlMjBiYXNlJTIwbW9kZWwlMEElMjAlMjAlMjAlMjBmb3IlMjBwYXJhbSUyMGluJTIwbW9kZWwucGFyYW1ldGVycygpJTNBJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwcGFyYW0ucmVxdWlyZXNfZ3JhZCUyMCUzRCUyMEZhbHNlJTBBJTIwJTIwJTIwJTIwZm9yJTIwcGFyYW0lMjBpbiUyMG1vZGVsLmdldF9jbGFzc2lmaWVyKCkucGFyYW1ldGVycygpJTNBJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwcGFyYW0ucmVxdWlyZXNfZ3JhZCUyMCUzRCUyMFRydWUlMEElMEElMjAlMjAlMjAlMjAlMjMlMjBZb3UlMjBjYW4lMjBub3JtYWxpemUlMjB0aGUlMjBiYXRjaGVzJTIwb2YlMjBpbWFnZXMlMjB0byUyMGJlJTIwYSUyMGJpdCUyMGZhc3RlciUwQSUyMCUyMCUyMCUyMG1lYW4lMjAlM0QlMjB0b3JjaC50ZW5zb3IobW9kZWwuZGVmYXVsdF9jZmclNUIlMjJtZWFuJTIyJTVEKSU1Qk5vbmUlMkMlMjAlM0ElMkMlMjBOb25lJTJDJTIwTm9uZSU1RCUwQSUyMCUyMCUyMCUyMHN0ZCUyMCUzRCUyMHRvcmNoLnRlbnNvcihtb2RlbC5kZWZhdWx0X2NmZyU1QiUyMnN0ZCUyMiU1RCklNUJOb25lJTJDJTIwJTNBJTJDJTIwTm9uZSUyQyUyME5vbmUlNUQlMEElMEElMjAlMjAlMjAlMjAlMjMlMjBUbyUyMG1ha2UlMjB0aGVzZSUyMGNvbnN0YW50cyUyMGF2YWlsYWJsZSUyMG9uJTIwdGhlJTIwYWN0aXZlJTIwZGV2aWNlJTJDJTIwc2V0JTIwaXQlMjB0byUyMHRoZSUyMGFjY2VsZXJhdG9yJTIwZGV2aWNlJTBBJTIwJTIwJTIwJTIwbWVhbiUyMCUzRCUyMG1lYW4udG8oYWNjZWxlcmF0b3IuZGV2aWNlKSUwQSUyMCUyMCUyMCUyMHN0ZCUyMCUzRCUyMHN0ZC50byhhY2NlbGVyYXRvci5kZXZpY2UpJTBBJTBBJTIwJTIwJTIwJTIwJTIzJTIwSW5zdGFudGlhdGUlMjB0aGUlMjBvcHRpbWl6ZXIlMEElMjAlMjAlMjAlMjBvcHRpbWl6ZXIlMjAlM0QlMjB0b3JjaC5vcHRpbS5BZGFtKHBhcmFtcyUzRG1vZGVsLnBhcmFtZXRlcnMoKSUyQyUyMGxyJTNEM2UtMiUyMCUyRiUyMDI1KSUwQSUwQSUyMCUyMCUyMCUyMCUyMyUyMEluc3RhbnRpYXRlJTIwdGhlJTIwbGVhcm5pbmclMjByYXRlJTIwc2NoZWR1bGVyJTBBJTIwJTIwJTIwJTIwbHJfc2NoZWR1bGVyJTIwJTNEJTIwT25lQ3ljbGVMUihvcHRpbWl6ZXIlM0RvcHRpbWl6ZXIlMkMlMjBtYXhfbHIlM0QzZS0yJTJDJTIwZXBvY2hzJTNENSUyQyUyMHN0ZXBzX3Blcl9lcG9jaCUzRGxlbih0cmFpbl9kYXRhbG9hZGVyKSklMEElMEElMjAlMjAlMjAlMjAlMjMlMjBQcmVwYXJlJTIwZXZlcnl0aGluZyUwQSUyMCUyMCUyMCUyMCUyMyUyMFRoZXJlJTIwaXMlMjBubyUyMHNwZWNpZmljJTIwb3JkZXIlMjB0byUyMHJlbWVtYmVyJTJDJTIweW91JTIwanVzdCUyMG5lZWQlMjB0byUyMHVucGFjayUyMHRoZSUyMG9iamVjdHMlMjBpbiUyMHRoZSUyMHNhbWUlMjBvcmRlciUyMHlvdSUyMGdhdmUlMjB0aGVtJTIwdG8lMjB0aGUlMEElMjAlMjAlMjAlMjAlMjMlMjBwcmVwYXJlJTIwbWV0aG9kLiUwQSUyMCUyMCUyMCUyMG1vZGVsJTJDJTIwb3B0aW1pemVyJTJDJTIwdHJhaW5fZGF0YWxvYWRlciUyQyUyMGV2YWxfZGF0YWxvYWRlciUyQyUyMGxyX3NjaGVkdWxlciUyMCUzRCUyMGFjY2VsZXJhdG9yLnByZXBhcmUoJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbW9kZWwlMkMlMjBvcHRpbWl6ZXIlMkMlMjB0cmFpbl9kYXRhbG9hZGVyJTJDJTIwZXZhbF9kYXRhbG9hZGVyJTJDJTIwbHJfc2NoZWR1bGVyJTBBJTIwJTIwJTIwJTIwKSUwQSUwQSUyMCUyMCUyMCUyMCUyMyUyME5vdyUyMHlvdSUyMHRyYWluJTIwdGhlJTIwbW9kZWwlMEElMjAlMjAlMjAlMjBmb3IlMjBlcG9jaCUyMGluJTIwcmFuZ2UoNSklM0ElMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBtb2RlbC50cmFpbigpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwZm9yJTIwYmF0Y2glMjBpbiUyMHRyYWluX2RhdGFsb2FkZXIlM0ElMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBpbnB1dHMlMjAlM0QlMjAoYmF0Y2glNUIlMjJpbWFnZSUyMiU1RCUyMC0lMjBtZWFuKSUyMCUyRiUyMHN0ZCUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMG91dHB1dHMlMjAlM0QlMjBtb2RlbChpbnB1dHMpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbG9zcyUyMCUzRCUyMHRvcmNoLm5uLmZ1bmN0aW9uYWwuY3Jvc3NfZW50cm9weShvdXRwdXRzJTJDJTIwYmF0Y2glNUIlMjJsYWJlbCUyMiU1RCklMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBhY2NlbGVyYXRvci5iYWNrd2FyZChsb3NzKSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMG9wdGltaXplci5zdGVwKCklMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBscl9zY2hlZHVsZXIuc3RlcCgpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwb3B0aW1pemVyLnplcm9fZ3JhZCgpJTBBJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbW9kZWwuZXZhbCgpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwYWNjdXJhdGUlMjAlM0QlMjAwJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbnVtX2VsZW1zJTIwJTNEJTIwMCUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGZvciUyMGJhdGNoJTIwaW4lMjBldmFsX2RhdGFsb2FkZXIlM0ElMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBpbnB1dHMlMjAlM0QlMjAoYmF0Y2glNUIlMjJpbWFnZSUyMiU1RCUyMC0lMjBtZWFuKSUyMCUyRiUyMHN0ZCUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMHdpdGglMjB0b3JjaC5ub19ncmFkKCklM0ElMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBvdXRwdXRzJTIwJTNEJTIwbW9kZWwoaW5wdXRzKSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMHByZWRpY3Rpb25zJTIwJTNEJTIwb3V0cHV0cy5hcmdtYXgoZGltJTNELTEpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwYWNjdXJhdGVfcHJlZHMlMjAlM0QlMjBhY2NlbGVyYXRvci5nYXRoZXIocHJlZGljdGlvbnMpJTIwJTNEJTNEJTIwYWNjZWxlcmF0b3IuZ2F0aGVyKGJhdGNoJTVCJTIybGFiZWwlMjIlNUQpJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwbnVtX2VsZW1zJTIwJTJCJTNEJTIwYWNjdXJhdGVfcHJlZHMuc2hhcGUlNUIwJTVEJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwYWNjdXJhdGUlMjAlMkIlM0QlMjBhY2N1cmF0ZV9wcmVkcy5sb25nKCkuc3VtKCklMEElMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjBldmFsX21ldHJpYyUyMCUzRCUyMGFjY3VyYXRlLml0ZW0oKSUyMCUyRiUyMG51bV9lbGVtcyUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMyUyMFVzZSUyMGFjY2VsZXJhdG9yLnByaW50JTIwdG8lMjBwcmludCUyMG9ubHklMjBvbiUyMHRoZSUyMG1haW4lMjBwcm9jZXNzLiUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMGFjY2VsZXJhdG9yLnByaW50KGYlMjJlcG9jaCUyMCU3QmVwb2NoJTdEJTNBJTIwJTdCMTAwJTIwKiUyMGV2YWxfbWV0cmljJTNBLjJmJTdEJTIyKQ==",highlighted:`<span class="hljs-keyword">def</span> <span class="hljs-title function_">training_loop</span>(<span class="hljs-params">mixed_precision=<span class="hljs-string">&quot;fp16&quot;</span>, seed: <span class="hljs-built_in">int</span> = <span class="hljs-number">42</span>, batch_size: <span class="hljs-built_in">int</span> = <span class="hljs-number">64</span></span>):
 set_seed(seed)
 <span class="hljs-comment"># Initialize accelerator</span>
 accelerator = Accelerator(mixed_precision=mixed_precision)
 <span class="hljs-comment"># Build dataloaders</span>
 train_dataloader, eval_dataloader = get_dataloaders(batch_size)
 <span class="hljs-comment"># Instantiate the model (you build the model here so that the seed also controls new weight initializations)</span>
 model = create_model(<span class="hljs-string">&quot;resnet50d&quot;</span>, pretrained=<span class="hljs-literal">True</span>, num_classes=<span class="hljs-built_in">len</span>(label_to_id))
 <span class="hljs-comment"># Freeze the base model</span>
 <span class="hljs-keyword">for</span> param <span class="hljs-keyword">in</span> model.parameters():
 param.requires_grad = <span class="hljs-literal">False</span>
 <span class="hljs-keyword">for</span> param <span class="hljs-keyword">in</span> model.get_classifier().parameters():
 param.requires_grad = <span class="hljs-literal">True</span>
 <span class="hljs-comment"># You can normalize the batches of images to be a bit faster</span>
 mean = torch.tensor(model.default_cfg[<span class="hljs-string">&quot;mean&quot;</span>])[<span class="hljs-literal">None</span>, :, <span class="hljs-literal">None</span>, <span class="hljs-literal">None</span>]
 std = torch.tensor(model.default_cfg[<span class="hljs-string">&quot;std&quot;</span>])[<span class="hljs-literal">None</span>, :, <span class="hljs-literal">None</span>, <span class="hljs-literal">None</span>]
 <span class="hljs-comment"># To make these constants available on the active device, set it to the accelerator device</span>
 mean = mean.to(accelerator.device)
 std = std.to(accelerator.device)
 <span class="hljs-comment"># Instantiate the optimizer</span>
 optimizer = torch.optim.Adam(params=model.parameters(), lr=<span class="hljs-number">3e-2</span> / <span class="hljs-number">25</span>)
 <span class="hljs-comment"># Instantiate the learning rate scheduler</span>
 lr_scheduler = OneCycleLR(optimizer=optimizer, max_lr=<span class="hljs-number">3e-2</span>, epochs=<span class="hljs-number">5</span>, steps_per_epoch=<span class="hljs-built_in">len</span>(train_dataloader))
 <span class="hljs-comment"># Prepare everything</span>
 <span class="hljs-comment"># There is no specific order to remember, you just need to unpack the objects in the same order you gave them to the</span>
 <span class="hljs-comment"># prepare method.</span>
 model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
 model, optimizer, train_dataloader, eval_dataloader, lr_scheduler
 )
 <span class="hljs-comment"># Now you train the model</span>
 <span class="hljs-keyword">for</span> epoch <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">5</span>):
 model.train()
 <span class="hljs-keyword">for</span> batch <span class="hljs-keyword">in</span> train_dataloader:
 inputs = (batch[<span class="hljs-string">&quot;image&quot;</span>] - mean) / std
 outputs = model(inputs)
 loss = torch.nn.functional.cross_entropy(outputs, batch[<span class="hljs-string">&quot;label&quot;</span>])
 accelerator.backward(loss)
 optimizer.step()
 lr_scheduler.step()
 optimizer.zero_grad()
 model.<span class="hljs-built_in">eval</span>()
 accurate = <span class="hljs-number">0</span>
 num_elems = <span class="hljs-number">0</span>
 <span class="hljs-keyword">for</span> batch <span class="hljs-keyword">in</span> eval_dataloader:
 inputs = (batch[<span class="hljs-string">&quot;image&quot;</span>] - mean) / std
 <span class="hljs-keyword">with</span> torch.no_grad():
 outputs = model(inputs)
 predictions = outputs.argmax(dim=-<span class="hljs-number">1</span>)
 accurate_preds = accelerator.gather(predictions) == accelerator.gather(batch[<span class="hljs-string">&quot;label&quot;</span>])
 num_elems += accurate_preds.shape[<span class="hljs-number">0</span>]
 accurate += accurate_preds.long().<span class="hljs-built_in">sum</span>()
 eval_metric = accurate.item() / num_elems
 <span class="hljs-comment"># Use accelerator.print to print only on the main process.</span>
 accelerator.<span class="hljs-built_in">print</span>(<span class="hljs-string">f&quot;epoch <span class="hljs-subst">{epoch}</span>: <span class="hljs-subst">{<span class="hljs-number">100</span> * eval_metric:<span class="hljs-number">.2</span>f}</span>&quot;</span>)`,lang:"python",wrap:!1});var D=l(x,2);c(D,{title:"Using the notebook_launcher",local:"using-the-notebooklauncher",headingTag:"h2"});var K=l(D,6);a(K,{code:"ZnJvbSUyMGFjY2VsZXJhdGUlMjBpbXBvcnQlMjBub3RlYm9va19sYXVuY2hlcg==",highlighted:'<span class="hljs-keyword">from</span> accelerate <span class="hljs-keyword">import</span> notebook_launcher',lang:"python",wrap:!1});var q=l(K,2);a(q,{code:"YXJncyUyMCUzRCUyMCglMjJmcDE2JTIyJTJDJTIwNDIlMkMlMjA2NCklMEFub3RlYm9va19sYXVuY2hlcih0cmFpbmluZ19sb29wJTJDJTIwYXJncyUyQyUyMG51bV9wcm9jZXNzZXMlM0QyKQ==",highlighted:`args = (<span class="hljs-string">&quot;fp16&quot;</span>, <span class="hljs-number">42</span>, <span class="hljs-number">64</span>)
notebook_launcher(training_loop, args, num_processes=<span class="hljs-number">2</span>)`,lang:"python",wrap:!1});var L=l(q,6);a(L,{code:"bm90ZWJvb2tfbGF1bmNoZXIodHJhaW5pbmdfbG9vcCUyQyUyMGFyZ3MlMkMlMjBtYXN0ZXJfYWRkciUzRCUyMjE3Mi4zMS40My44JTIyJTJDJTIwbm9kZV9yYW5rJTNEMCUyQyUyMG51bV9ub2RlcyUzRDIlMkMlMjBudW1fcHJvY2Vzc2VzJTNEOCk=",highlighted:'notebook_launcher(training_loop, args, master_addr=<span class="hljs-string">&quot;172.31.43.8&quot;</span>, node_rank=<span class="hljs-number">0</span>, num_nodes=<span class="hljs-number">2</span>, num_processes=<span class="hljs-number">8</span>)',lang:"python",wrap:!1});var P=l(L,4);r(P,{children:(s,n)=>{var e=Zl();t(s,e)},$$slots:{default:!0}});var $=l(P,2);a($,{code:"bm90ZWJvb2tfbGF1bmNoZXIodHJhaW5pbmdfbG9vcCUyQyUyMGFyZ3MlMkMlMjBtYXN0ZXJfYWRkciUzRCUyMjE3Mi4zMS40My44JTIyJTJDJTIwbm9kZV9yYW5rJTNEMSUyQyUyMG51bV9ub2RlcyUzRDIlMkMlMjBudW1fcHJvY2Vzc2VzJTNEOCk=",highlighted:'notebook_launcher(training_loop, args, master_addr=<span class="hljs-string">&quot;172.31.43.8&quot;</span>, node_rank=<span class="hljs-number">1</span>, num_nodes=<span class="hljs-number">2</span>, num_processes=<span class="hljs-number">8</span>)',lang:"python",wrap:!1});var O=l($,4);a(O,{code:"bW9kZWwlMjAlM0QlMjBjcmVhdGVfbW9kZWwoJTIycmVzbmV0NTBkJTIyJTJDJTIwcHJldHJhaW5lZCUzRFRydWUlMkMlMjBudW1fY2xhc3NlcyUzRGxlbihsYWJlbF90b19pZCkpJTBBJTBBYXJncyUyMCUzRCUyMChtb2RlbCUyQyUyMCUyMmZwMTYlMjIlMkMlMjA0MiUyQyUyMDY0KSUwQW5vdGVib29rX2xhdW5jaGVyKHRyYWluaW5nX2xvb3AlMkMlMjBhcmdzJTJDJTIwbnVtX3Byb2Nlc3NlcyUzRDgp",highlighted:`model = create_model(<span class="hljs-string">&quot;resnet50d&quot;</span>, pretrained=<span class="hljs-literal">True</span>, num_classes=<span class="hljs-built_in">len</span>(label_to_id))
args = (model, <span class="hljs-string">&quot;fp16&quot;</span>, <span class="hljs-number">42</span>, <span class="hljs-number">64</span>)
notebook_launcher(training_loop, args, num_processes=<span class="hljs-number">8</span>)`,lang:"python",wrap:!1});var ll=l(O,4);a(ll,{code:"bm90ZWJvb2tfbGF1bmNoZXIoJTBBJTIwJTIwJTIwJTIwdHJhaW5pbmdfbG9vcCUyQyUwQSUyMCUyMCUyMCUyMGFyZ3MlMkMlMEElMjAlMjAlMjAlMjBudW1fcHJvY2Vzc2VzJTNEMiUyQyUwQSUyMCUyMCUyMCUyMG1heF9yZXN0YXJ0cyUzRDMlMEEp",highlighted:`notebook_launcher(
 training_loop,
 args,
 num_processes=<span class="hljs-number">2</span>,
 max_restarts=<span class="hljs-number">3</span>
)`,lang:"python",wrap:!1});var al=l(ll,4);a(al,{code:"TGF1bmNoaW5nJTIwdHJhaW5pbmclMjBvbiUyMDIlMjBHUFVzLiUwQWVwb2NoJTIwMCUzQSUyMDg4LjEyJTBBZXBvY2glMjAxJTNBJTIwOTEuNzMlMEFlcG9jaCUyMDIlM0ElMjA5Mi41OCUwQWVwb2NoJTIwMyUzQSUyMDkzLjkwJTBBZXBvY2glMjA0JTNBJTIwOTQuNzE=",highlighted:`Launching training on <span class="hljs-number">2</span> GPUs.
epoch <span class="hljs-number">0</span>: <span class="hljs-number">88.12</span>
epoch <span class="hljs-number">1</span>: <span class="hljs-number">91.73</span>
epoch <span class="hljs-number">2</span>: <span class="hljs-number">92.58</span>
epoch <span class="hljs-number">3</span>: <span class="hljs-number">93.90</span>
epoch <span class="hljs-number">4</span>: <span class="hljs-number">94.71</span>`,lang:"python",wrap:!1});var sl=l(al,6);a(sl,{code:"YWNjZWxlcmF0ZSUyMGxhdW5jaA==",highlighted:"accelerate launch",lang:"bash",wrap:!1});var el=l(sl,2);c(el,{title:"Debugging",local:"debugging",headingTag:"h2"});var nl=l(el,4);c(nl,{title:"Conclusion",local:"conclusion",headingTag:"h2"});var cl=l(nl,6);il(cl,{source:"https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/notebook.md"}),Tl(2),t(tl,p),wl()}export{kl as component};

Xet Storage Details

Size:
48.4 kB
·
Xet hash:
5e0a5dd610f01ef90e09d6f313b45250c823ea868c1c8fe0b4aa71d527f13980

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.