Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Fine-grained FP8","local":"fine-grained-fp8","sections":[{"title":"DeepGEMM fast path","local":"deepgemm-fast-path","sections":[],"depth":2},{"title":"UE8M0 scale format","local":"ue8m0-scale-format","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_39895/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/entry/start.ed44d8d9.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/scheduler.31fdf58d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/singletons.6a6316d3.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/index.252883d5.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/paths.6325da18.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/entry/app.75a9946c.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/preload-helper.f369a5fb.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/index.2f76fdf0.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/nodes/0.4d11f797.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/nodes/638.9b3126f1.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/CopyLLMTxtMenu.121ef535.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/MermaidChart.svelte_svelte_type_style_lang.92a7ab29.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/IconCopy.ac192424.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_39895/en/_app/immutable/chunks/CodeBlock.e52df5d6.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Fine-grained FP8","local":"fine-grained-fp8","sections":[{"title":"DeepGEMM fast path","local":"deepgemm-fast-path","sections":[],"depth":2},{"title":"UE8M0 scale format","local":"ue8m0-scale-format","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"><div class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg></span> <span>Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg></button></div> </div> <h1 class="relative group"><a id="fine-grained-fp8" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#fine-grained-fp8"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Fine-grained FP8</span></h1> <p data-svelte-h="svelte-hrrhz7">Fine-grained FP8 quantization quantizes the weights and activations to fp8.</p> <ul data-svelte-h="svelte-1rx58b3"><li>The weights are quantized to 8-bits for each 2D block (<code>weight_block_size=(128, 128)</code>).</li> <li>The activations are quantized to 8-bits for each group per token. The group value matches the weights in the input channel (128 by default).</li></ul> <p data-svelte-h="svelte-10oa2n0">FP8 quantization enables support for <a href="https://hf.co/papers/2412.19437" rel="nofollow">DeepSeek-V3</a> and DeepSeek-R1.</p> <div class="flex justify-center" data-svelte-h="svelte-cd82ay"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/b7b3b34bf826a6423ea82ffc57ecac80c46c3c76/transformers/quantization/quantization_deepseek.png"></div> <blockquote class="tip" data-svelte-h="svelte-20otzg"><p>You need a GPU with Compute Capability>=9 (H100), and install a PyTorch version compatible with the CUDA version of your GPU.</p></blockquote> <p data-svelte-h="svelte-1rty8u8">Install Accelerate and upgrade to the latest version of PyTorch.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-bash "><!-- HTML_TAG_START -->pip install --upgrade accelerate torch<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1r0rwxu">Create a <a href="/docs/transformers/pr_39895/en/main_classes/quantization#transformers.FineGrainedFP8Config">FineGrainedFP8Config</a> class and pass it to <a href="/docs/transformers/pr_39895/en/main_classes/model#transformers.PreTrainedModel.from_pretrained">from_pretrained()</a> to quantize it. The weights are loaded in full precision (<code>torch.float32</code>) by default regardless of the actual data type the weights are stored in. Set <code>dtype="auto"</code> to load the weights in the data type defined in a models <code>config.json</code> file to automatically load the most memory-optimal data type.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-py "><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> FineGrainedFP8Config, AutoModelForCausalLM, AutoTokenizer | |
| model_name = <span class="hljs-string">"meta-llama/Meta-Llama-3-8B"</span> | |
| quantization_config = FineGrainedFP8Config() | |
| quantized_model = AutoModelForCausalLM.from_pretrained(model_name, dtype=<span class="hljs-string">"auto"</span>, device_map=<span class="hljs-string">"auto"</span>, quantization_config=quantization_config) | |
| tokenizer = AutoTokenizer.from_pretrained(model_name) | |
| input_text = <span class="hljs-string">"What are we having for dinner?"</span> | |
| input_ids = tokenizer(input_text, return_tensors=<span class="hljs-string">"pt"</span>).to(quantized_model.device.<span class="hljs-built_in">type</span>) | |
| output = quantized_model.generate(**input_ids, max_new_tokens=<span class="hljs-number">10</span>) | |
| <span class="hljs-built_in">print</span>(tokenizer.decode(output[<span class="hljs-number">0</span>], skip_special_tokens=<span class="hljs-literal">True</span>))<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1xnxe1b">Use <a href="/docs/transformers/pr_39895/en/main_classes/model#transformers.PreTrainedModel.save_pretrained">save_pretrained()</a> to save the quantized model and reload it with <a href="/docs/transformers/pr_39895/en/main_classes/model#transformers.PreTrainedModel.from_pretrained">from_pretrained()</a>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-py "><!-- HTML_TAG_START -->quant_path = <span class="hljs-string">"/path/to/save/quantized/model"</span> | |
| model.save_pretrained(quant_path) | |
| model = AutoModelForCausalLM.from_pretrained(quant_path, device_map=<span class="hljs-string">"auto"</span>)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="deepgemm-fast-path" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#deepgemm-fast-path"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>DeepGEMM fast path</span></h2> <p data-svelte-h="svelte-flauiz">On Hopper (SM90+) and Blackwell (SM100+) GPUs, every FP8 linear automatically dispatches to the <a href="https://github.com/deepseek-ai/DeepGEMM" rel="nofollow">DeepGEMM</a> kernels from <a href="https://huggingface.co/kernels-community/deep-gemm" rel="nofollow">kernels-community/deep-gemm</a> when <code>weight_block_size=(128, 128)</code> and <code>activation_scheme="dynamic"</code>. DeepGEMM is 3-6x faster than the Triton fallback. Install or upgrade the <a href="https://github.com/huggingface/kernels" rel="nofollow">kernels</a> package to enable it.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-bash "><!-- HTML_TAG_START -->pip install -U kernels<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-j7cmdz">DeepGEMM JIT-compiles its kernels, so the CUDA toolchain (<code>nvcc</code>/<code>nvrtc</code>) must be available. The required CUDA runtime depends on the hardware, 12.3+ on Hopper and 12.9+ on Blackwell.</p> <p data-svelte-h="svelte-xlhaw9">If the kernel cannot load (missing <code>kernels</code>, unsupported GPU, missing CUDA toolchain, or older CUDA), Transformers logs a warning once and falls back to the Triton finegrained-fp8 kernel. Static activation quantization always stays on the Triton path.</p> <p data-svelte-h="svelte-59tdxk">To force the Triton fallback even when DeepGEMM is available, set <code>TRANSFORMERS_DISABLE_DEEPGEMM_LINEAR=1</code>. This only affects the FP8 linear dispatch and leaves the <code>"deepgemm"</code> experts backend untouched, which you switch with <a href="/docs/transformers/pr_39895/en/main_classes/model#transformers.PreTrainedModel.set_experts_implementation">set_experts_implementation()</a>.</p> <p data-svelte-h="svelte-9aq1io">For MoE experts, the DeepGEMM path is opt-in. Pass <code>experts_implementation="deepgemm"</code> (or <code>"deepgemm_megamoe"</code> on Blackwell) at load time to route the expert matmuls through DeepGEMM. See the <a href="../experts_interface">Experts backends</a> guide for the full set of options.</p> <h2 class="relative group"><a id="ue8m0-scale-format" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#ue8m0-scale-format"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>UE8M0 scale format</span></h2> <p data-svelte-h="svelte-1vhdabc">DeepSeek V4-style checkpoints store FP8 weight scales in the packed <code>float8_e8m0fnu</code> format instead of <code>float32</code>. These checkpoints are pre-quantized and set <code>scale_fmt="ue8m0"</code> in their quantization config. Both the DeepGEMM and Triton kernels read UE8M0 scales, so these checkpoints run on either path.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/quantization/finegrained_fp8.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg> <span data-svelte-h="svelte-zjs2n5"><span class="underline">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_sw4oup = { | |
| assets: "/docs/transformers/pr_39895/en", | |
| base: "/docs/transformers/pr_39895/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_39895/en/_app/immutable/entry/start.ed44d8d9.js"), | |
| import("/docs/transformers/pr_39895/en/_app/immutable/entry/app.75a9946c.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 638], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 20.1 kB
- Xet hash:
- 4bd5e2cc82064bc6b7af98b8ffa409d1e1cbf8cc52f231defdc6c60c71baa2a8
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.