accelerate/main/en/usage_guides/mps.html

<meta charset="utf-8" /><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Accelerated PyTorch Training on Mac&quot;,&quot;local&quot;:&quot;accelerated-pytorch-training-on-mac&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Benefits of Training and Inference using Apple Silicon Chips&quot;,&quot;local&quot;:&quot;benefits-of-training-and-inference-using-apple-silicon-chips&quot;,&quot;sections&quot;:[],&quot;depth&quot;:3},{&quot;title&quot;:&quot;How it works out of the box&quot;,&quot;local&quot;:&quot;how-it-works-out-of-the-box&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;A few caveats to be aware of&quot;,&quot;local&quot;:&quot;a-few-caveats-to-be-aware-of&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}">
		<link href="/docs/accelerate/main/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/entry/start.2ea03080.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/chunks/scheduler.defa9a21.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/chunks/singletons.aff0b9fc.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/chunks/index.beade68d.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/chunks/paths.2c85d1a6.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/entry/app.e6812672.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/chunks/index.fe795e71.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/nodes/0.39c84d5d.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/chunks/each.e59479a4.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/nodes/48.82536b9a.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/chunks/CodeBlock.42404125.js">
		<link rel="modulepreload" href="/docs/accelerate/main/en/_app/immutable/chunks/EditOnGithub.0f575778.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Accelerated PyTorch Training on Mac&quot;,&quot;local&quot;:&quot;accelerated-pytorch-training-on-mac&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Benefits of Training and Inference using Apple Silicon Chips&quot;,&quot;local&quot;:&quot;benefits-of-training-and-inference-using-apple-silicon-chips&quot;,&quot;sections&quot;:[],&quot;depth&quot;:3},{&quot;title&quot;:&quot;How it works out of the box&quot;,&quot;local&quot;:&quot;how-it-works-out-of-the-box&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;A few caveats to be aware of&quot;,&quot;local&quot;:&quot;a-few-caveats-to-be-aware-of&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}"><!-- HEAD_svelte-u9bgzb_END -->      <p></p>   <h1 class="relative group"><a id="accelerated-pytorch-training-on-mac" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#accelerated-pytorch-training-on-mac"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Accelerated PyTorch Training on Mac</span></h1> <p data-svelte-h="svelte-1mfz1q4">With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training.
This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
Apple’s Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new <code>&quot;mps&quot;</code> device.
This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
For more information please refer official documents <a href="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/" rel="nofollow">Introducing Accelerated PyTorch Training on Mac</a>
and <a href="https://pytorch.org/docs/stable/notes/mps.html" rel="nofollow">MPS BACKEND</a>.</p>  <h3 class="relative group"><a id="benefits-of-training-and-inference-using-apple-silicon-chips" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#benefits-of-training-and-inference-using-apple-silicon-chips"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Benefits of Training and Inference using Apple Silicon Chips</span></h3> <ol data-svelte-h="svelte-oq3rck"><li>Enables users to train larger networks or batch sizes locally</li> <li>Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture.
Therefore, improving end-to-end performance.</li> <li>Reduces costs associated with cloud-based development or the need for additional local GPUs.</li></ol> <p data-svelte-h="svelte-1kbpcbh"><strong>Pre-requisites</strong>: To install torch with mps support,
please follow this nice medium article <a href="https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1" rel="nofollow">GPU-Acceleration Comes to PyTorch on M1 Macs</a>.</p>  <h2 class="relative group"><a id="how-it-works-out-of-the-box" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#how-it-works-out-of-the-box"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>How it works out of the box</span></h2> <p data-svelte-h="svelte-1lubatz">It is enabled by default on MacOs machines with MPS enabled Apple Silicon GPUs.
To disable it, pass <code>--cpu</code> flag to <code>accelerate launch</code> command or answer the corresponding question when answering the <code>accelerate config</code> questionnaire.</p> <p data-svelte-h="svelte-1k89y14">You can directly run the following script to test it out on MPS enabled Apple Silicon machines:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->accelerate launch /examples/cv_example.py --data_dir images<!-- HTML_TAG_END --></pre></div>  <h2 class="relative group"><a id="a-few-caveats-to-be-aware-of" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#a-few-caveats-to-be-aware-of"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>A few caveats to be aware of</span></h2> <ol data-svelte-h="svelte-9odwh8"><li>We strongly recommend to install PyTorch &gt;= 1.13 (nightly version at the time of writing) on your MacOS machine.
It has major fixes related to model correctness and performance improvements for transformer based models.
Please refer to <a href="https://github.com/pytorch/pytorch/issues/82707" rel="nofollow">https://github.com/pytorch/pytorch/issues/82707</a> for more details.</li> <li>Distributed setups <code>gloo</code> and <code>nccl</code> are not working with <code>mps</code> device.
This means that currently only single GPU of <code>mps</code> device type can be used.</li></ol> <p data-svelte-h="svelte-ggojqf">Finally, please, remember that, 🤗 <code>Accelerate</code> only integrates MPS backend, therefore if you
have any problems or questions with regards to MPS backend usage, please, file an issue with <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow">PyTorch GitHub</a>.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/mps.md" target="_blank"><span data-svelte-h="svelte-1kd6by1">&lt;</span> <span data-svelte-h="svelte-x0xyl0">&gt;</span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a>  <p></p> 
			
			<script>
				{
					__sveltekit_1fyccrg = {
						assets: "/docs/accelerate/main/en",
						base: "/docs/accelerate/main/en",
						env: {}
					};

					const element = document.currentScript.parentElement;

					const data = [null,null];

					Promise.all([
						import("/docs/accelerate/main/en/_app/immutable/entry/start.2ea03080.js"),
						import("/docs/accelerate/main/en/_app/immutable/entry/app.e6812672.js")
					]).then(([kit, app]) => {
						kit.start(app, element, {
							node_ids: [0, 48],
							data,
							form: null,
							error: null
						});
					});
				}
			</script>