Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Inference","local":"inference","sections":[{"title":"Loading","local":"loading","sections":[{"title":"Transformers models","local":"transformers-models","sections":[],"depth":3},{"title":"Diffusers models","local":"diffusers-models","sections":[],"depth":3}],"depth":2},{"title":"Compilation","local":"compilation","sections":[],"depth":2},{"title":"Static shape","local":"static-shape","sections":[],"depth":2},{"title":"Configuration","local":"configuration","sections":[],"depth":2},{"title":"Weight quantization","local":"weight-quantization","sections":[],"depth":2}],"depth":1}"/> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/entry/start.CatJ25dz.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/B-VqpFX9.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/Cbbc2wwc.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/entry/app.KpdgG_wN.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/DeDI0Q53.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/6YgYxE4i.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/B9oc0ti6.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/nodes/0.CgngmqY0.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/CczWW-MX.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/nodes/5.BHg_WzKM.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/yrm_jVWY.js" rel="modulepreload"> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/chunks/CHJ7u4WY.js" rel="modulepreload"> | |
| <!--1lahsqv--><meta name="hf:doc:metadata" content="{"title":"Inference","local":"inference","sections":[{"title":"Loading","local":"loading","sections":[{"title":"Transformers models","local":"transformers-models","sections":[],"depth":3},{"title":"Diffusers models","local":"diffusers-models","sections":[],"depth":3}],"depth":2},{"title":"Compilation","local":"compilation","sections":[],"depth":2},{"title":"Static shape","local":"static-shape","sections":[],"depth":2},{"title":"Configuration","local":"configuration","sections":[],"depth":2},{"title":"Weight quantization","local":"weight-quantization","sections":[],"depth":2}],"depth":1}"/><!----> | |
| <link href="/docs/optimum.intel/pr_1908/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"><div class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----></span> <span>Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg><!----></button></div> <!--[-1--><!--]--></div><!----> <!--[0--><h1 class="relative group"><a id="inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#inference"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Inference</span></h1><!--]--><!----> <p>Optimum Intel can be used to load optimized models from the <a href="https://huggingface.co/models?library=openvino&sort=downloads" rel="nofollow">Hub</a> and create pipelines to run inference with OpenVINO Runtime on a variety of Intel processors (<a href="https://docs.openvino.ai/2024/about-openvino/compatibility-and-support/supported-devices.html" rel="nofollow">see</a> the full list of supported devices)</p> <!--[1--><h2 class="relative group"><a id="loading" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#loading"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Loading</span></h2><!--]--><!----> <!--[2--><h3 class="relative group"><a id="transformers-models" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#transformers-models"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Transformers models</span></h3><!--]--><!----> <p>Once <a href="export">your model was exported</a>, you can load it by replacing the <code>AutoModelForXxx</code> class with the corresponding <code>OVModelForXxx</code>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-diff "><!----><span class="hljs-deletion">- from transformers import AutoModelForCausalLM</span> | |
| <span class="hljs-addition">+ from optimum.intel import OVModelForCausalLM</span> | |
| from transformers import AutoTokenizer, pipeline | |
| model_id = "helenai/gpt2-ov" | |
| <span class="hljs-deletion">- model = AutoModelForCausalLM.from_pretrained(model_id)</span> | |
| # here the model was already exported so no need to set export=True | |
| <span class="hljs-addition">+ model = OVModelForCausalLM.from_pretrained(model_id)</span> | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| pipe = pipeline("text-generation", model=model, tokenizer=tokenizer) | |
| results = pipe("He's a dreadful magician and")<!----></pre></div><!----> <p>As shown in the table below, each task is associated with a class enabling to automatically load your model.</p> <table><thead><tr><th>Auto Class</th><th>Task</th></tr></thead><tbody><tr><td><code>OVModelForSequenceClassification</code></td><td><code>text-classification</code></td></tr><tr><td><code>OVModelForTokenClassification</code></td><td><code>token-classification</code></td></tr><tr><td><code>OVModelForQuestionAnswering</code></td><td><code>question-answering</code></td></tr><tr><td><code>OVModelForAudioClassification</code></td><td><code>audio-classification</code></td></tr><tr><td><code>OVModelForImageClassification</code></td><td><code>image-classification</code></td></tr><tr><td><code>OVModelForFeatureExtraction</code></td><td><code>feature-extraction</code></td></tr><tr><td><code>OVModelForMaskedLM</code></td><td><code>fill-mask</code></td></tr><tr><td><code>OVModelForCausalLM</code></td><td><code>text-generation-with-past</code></td></tr><tr><td><code>OVModelForSeq2SeqLM</code></td><td><code>text2text-generation-with-past</code></td></tr><tr><td><code>OVModelForSpeechSeq2Seq</code></td><td><code>automatic-speech-recognition</code></td></tr><tr><td><code>OVModelForImageTextToText</code></td><td><code>image-to-text</code></td></tr><tr><td><code>OVModelForTextToSpeechSeq2Seq</code></td><td><code>text-to-audio</code></td></tr></tbody></table> <!--[2--><h3 class="relative group"><a id="diffusers-models" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#diffusers-models"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Diffusers models</span></h3><!--]--><!----> <p>Make sure you have 🤗 Diffusers installed. To install <code>diffusers</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-bash "><!---->pip install diffusers<!----></pre></div><!----> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-diff "><!----><span class="hljs-deletion">- from diffusers import StableDiffusionPipeline</span> | |
| <span class="hljs-addition">+ from optimum.intel import OVStableDiffusionPipeline</span> | |
| model_id = "echarlaix/stable-diffusion-v1-5-openvino" | |
| <span class="hljs-deletion">- pipeline = StableDiffusionPipeline.from_pretrained(model_id)</span> | |
| <span class="hljs-addition">+ pipeline = OVStableDiffusionPipeline.from_pretrained(model_id)</span> | |
| prompt = "sailing ship in storm by Rembrandt" | |
| images = pipeline(prompt).images<!----></pre></div><!----> <p>As shown in the table below, each task is associated with a class enabling to automatically load your model.</p> <table><thead><tr><th>Auto Class</th><th>Task</th></tr></thead><tbody><tr><td><code>OVStableDiffusionPipeline</code></td><td><code>text-to-image</code></td></tr><tr><td><code>OVStableDiffusionImg2ImgPipeline</code></td><td><code>image-to-image</code></td></tr><tr><td><code>OVStableDiffusionInpaintPipeline</code></td><td><code>inpaint</code></td></tr><tr><td><code>OVStableDiffusionXLPipeline</code></td><td><code>text-to-image</code></td></tr><tr><td><code>OVStableDiffusionXLImg2ImgPipeline</code></td><td><code>image-to-image</code></td></tr><tr><td><code>OVLatentConsistencyModelPipeline</code></td><td><code>text-to-image</code></td></tr><tr><td><code>OVLTXPipeline</code></td><td><code>text-to-video</code></td></tr><tr><td><code>OVPipelineForText2Video</code></td><td><code>text-to-video</code></td></tr><tr><td><code>OVLTXImageToVideoPipeline</code></td><td><code>image-to-video</code></td></tr><tr><td><code>OVPipelineForImage2Video</code></td><td><code>image-to-video</code></td></tr></tbody></table> <p>See the <a href="reference">reference documentation</a> for more information about parameters, and examples for different tasks.</p> <!--[1--><h2 class="relative group"><a id="compilation" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#compilation"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Compilation</span></h2><!--]--><!----> <p>By default the model will be compiled when instantiating an <code>OVModel</code>. In the case where the model is reshaped or placed to another device, the model will need to be recompiled again, which will happen by default before the first inference (thus inflating the latency of the first inference). To avoid an unnecessary compilation, you can disable the first compilation by setting <code>compile=False</code>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">from</span> optimum.intel <span class="hljs-keyword">import</span> OVModelForQuestionAnswering | |
| model_id = <span class="hljs-string">"distilbert/distilbert-base-cased-distilled-squad"</span> | |
| <span class="hljs-comment"># Load the model and disable the model compilation</span> | |
| model = OVModelForQuestionAnswering.from_pretrained(model_id, <span class="hljs-built_in">compile</span>=<span class="hljs-literal">False</span>)<!----></pre></div><!----> <p>To run inference on Intel integrated or discrete GPU, use <code>.to("gpu")</code>. On GPU, models run in FP16 precision by default. (See <a href="https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html" rel="nofollow">OpenVINO documentation</a> about installing drivers for GPU inference).</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!---->model.to(<span class="hljs-string">"gpu"</span>)<!----></pre></div><!----> <p>The model can be compiled:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!---->model.<span class="hljs-built_in">compile</span>()<!----></pre></div><!----> <!--[1--><h2 class="relative group"><a id="static-shape" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#static-shape"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Static shape</span></h2><!--]--><!----> <p>By default, dynamic shapes are supported, enabling inference for inputs of every shape. To speed up inference, static shapes can be enabled by giving the desired input shapes with <a href="reference#optimum.intel.OVBaseModel.reshape">.reshape()</a>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-comment"># Fix the batch size to 1 and the sequence length to 40</span> | |
| batch_size, seq_len = <span class="hljs-number">1</span>, <span class="hljs-number">40</span> | |
| model.reshape(batch_size, seq_len)<!----></pre></div><!----> <p>When fixing the shapes with the <code>reshape()</code> method, inference cannot be performed with an input of a different shape.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----> | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoTokenizer | |
| <span class="hljs-keyword">from</span> optimum.intel <span class="hljs-keyword">import</span> OVModelForQuestionAnswering | |
| model_id = <span class="hljs-string">"distilbert/distilbert-base-cased-distilled-squad"</span> | |
| model = OVModelForQuestionAnswering.from_pretrained(model_id, <span class="hljs-built_in">compile</span>=<span class="hljs-literal">False</span>) | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| batch_size, seq_len = <span class="hljs-number">1</span>, <span class="hljs-number">40</span> | |
| model.reshape(batch_size, seq_len) | |
| <span class="hljs-comment"># Compile the model before the first inference</span> | |
| model.<span class="hljs-built_in">compile</span>() | |
| question = <span class="hljs-string">"Which name is also used to describe the Amazon rainforest ?"</span> | |
| context = <span class="hljs-string">"The Amazon rainforest, also known as Amazonia or the Amazon Jungle"</span> | |
| tokens = tokenizer(question, context, max_length=seq_len, padding=<span class="hljs-string">"max_length"</span>, return_tensors=<span class="hljs-string">"np"</span>) | |
| outputs = model(**tokens)<!----></pre></div><!----> <p>For models that handle images, you can also specify the <code>height</code> and <code>width</code> when reshaping your model:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!---->batch_size, num_images, height, width = <span class="hljs-number">1</span>, <span class="hljs-number">1</span>, <span class="hljs-number">512</span>, <span class="hljs-number">512</span> | |
| pipeline.reshape(batch_size=batch_size, height=height, width=width, num_images_per_prompt=num_images) | |
| images = pipeline(prompt, height=height, width=width, num_images_per_prompt=num_images).images<!----></pre></div><!----> <!--[1--><h2 class="relative group"><a id="configuration" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#configuration"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Configuration</span></h2><!--]--><!----> <p>The <code>ov_config</code> parameter allow to provide custom OpenVINO configuration values. This can be used for example to enable full precision inference on devices where FP16 or BF16 inference precision is used by default.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!---->ov_config = {<span class="hljs-string">"INFERENCE_PRECISION_HINT"</span>: <span class="hljs-string">"f32"</span>} | |
| model = OVModelForSequenceClassification.from_pretrained(model_id, ov_config=ov_config)<!----></pre></div><!----> <p>Optimum Intel leverages OpenVINO’s model caching to speed up model compiling on GPU. By default a <code>model_cache</code> directory is created in the model’s directory in the <a href="https://huggingface.co/docs/huggingface_hub/main/en/guides/manage-cache" rel="nofollow">Hugging Face Hub cache</a>. To override this, use the ov_config parameter and set <code>CACHE_DIR</code> to a different value. To disable model caching on GPU, set <code>CACHE_DIR</code> to an empty string.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!---->ov_config = {<span class="hljs-string">"CACHE_DIR"</span>: <span class="hljs-string">""</span>} | |
| model = OVModelForSequenceClassification.from_pretrained(model_id, device=<span class="hljs-string">"gpu"</span>, ov_config=ov_config)<!----></pre></div><!----> <!--[1--><h2 class="relative group"><a id="weight-quantization" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#weight-quantization"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Weight quantization</span></h2><!--]--><!----> <p>You can also apply fp16, 8-bit or 4-bit weight compression on the Linear, Convolutional and Embedding layers when loading your model to reduce the memory footprint and inference latency.</p> <p>For more information on the quantization parameters checkout the <a href="optimization#weight-only-quantization">documentation</a>.</p> <blockquote class="warning"><p>If not specified, <code>load_in_8bit</code> will be set to <code>True</code> by default when models larger than 1 billion parameters are exported to the OpenVINO format (with <code>export=True</code>). You can disable it with <code>load_in_8bit=False</code>.</p><!----></blockquote><!----> <p>It’s also possible to apply quantization on both weights and activations using the <a href="optimization#static-quantization"><code>OVQuantizer</code></a>.</p> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]--> | |
| <script> | |
| { | |
| __sveltekit_1ofgn6n = { | |
| base: "/docs/optimum.intel/pr_1908/en", | |
| assets: "/docs/optimum.intel/pr_1908/en" | |
| }; | |
| const element = document.currentScript.parentElement; | |
| Promise.all([ | |
| import("/docs/optimum.intel/pr_1908/en/_app/immutable/entry/start.CatJ25dz.js"), | |
| import("/docs/optimum.intel/pr_1908/en/_app/immutable/entry/app.KpdgG_wN.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 5], | |
| data: [null,null], | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 39 kB
- Xet hash:
- 260be769ac614521d51922e11431d355d4eb98e126cc4a01b8eb40bd3a715e54
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.