Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Pipelines for inference","local":"pipelines-for-inference","sections":[{"title":"Pipeline usage","local":"pipeline-usage","sections":[],"depth":2},{"title":"Parameters","local":"parameters","sections":[{"title":"Device","local":"device","sections":[],"depth":3},{"title":"Batch size","local":"batch-size","sections":[],"depth":3},{"title":"Task specific parameters","local":"task-specific-parameters","sections":[],"depth":3}],"depth":2},{"title":"Using pipelines on a dataset","local":"using-pipelines-on-a-dataset","sections":[],"depth":2},{"title":"Using pipelines for a webserver","local":"using-pipelines-for-a-webserver","sections":[],"depth":2},{"title":"Vision pipeline","local":"vision-pipeline","sections":[],"depth":2},{"title":"Text pipeline","local":"text-pipeline","sections":[],"depth":2},{"title":"Multimodal pipeline","local":"multimodal-pipeline","sections":[],"depth":2},{"title":"Using pipeline on large models with 🤗 accelerate :","local":"using-pipeline-on-large-models-with--accelerate-","sections":[],"depth":2},{"title":"Creating web demos from pipelines with gradio","local":"creating-web-demos-from-pipelines-with-gradio","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_33913/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/entry/start.b67f883f.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/scheduler.25b97de1.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/singletons.62a184e0.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/index.e188933d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/paths.51881b9e.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/entry/app.e436b1f2.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/index.d9030fc9.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/nodes/0.05e395f5.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/nodes/392.f51f3950.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/Tip.baa67368.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/CodeBlock.e6cd0d95.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_33913/en/_app/immutable/chunks/EditOnGithub.91d95064.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Pipelines for inference","local":"pipelines-for-inference","sections":[{"title":"Pipeline usage","local":"pipeline-usage","sections":[],"depth":2},{"title":"Parameters","local":"parameters","sections":[{"title":"Device","local":"device","sections":[],"depth":3},{"title":"Batch size","local":"batch-size","sections":[],"depth":3},{"title":"Task specific parameters","local":"task-specific-parameters","sections":[],"depth":3}],"depth":2},{"title":"Using pipelines on a dataset","local":"using-pipelines-on-a-dataset","sections":[],"depth":2},{"title":"Using pipelines for a webserver","local":"using-pipelines-for-a-webserver","sections":[],"depth":2},{"title":"Vision pipeline","local":"vision-pipeline","sections":[],"depth":2},{"title":"Text pipeline","local":"text-pipeline","sections":[],"depth":2},{"title":"Multimodal pipeline","local":"multimodal-pipeline","sections":[],"depth":2},{"title":"Using pipeline on large models with 🤗 accelerate :","local":"using-pipeline-on-large-models-with--accelerate-","sections":[],"depth":2},{"title":"Creating web demos from pipelines with gradio","local":"creating-web-demos-from-pipelines-with-gradio","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="pipelines-for-inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#pipelines-for-inference"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Pipelines for inference</span></h1> <p data-svelte-h="svelte-1gmuqqg">The <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> makes it simple to use any model from the <a href="https://huggingface.co/models" rel="nofollow">Hub</a> for inference on any language, computer vision, speech, and multimodal tasks. Even if you don’t have experience with a specific modality or aren’t familiar with the underlying code behind the models, you can still use them for inference with the <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a>! This tutorial will teach you to:</p> <ul data-svelte-h="svelte-1l2zsnk"><li>Use a <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> for inference.</li> <li>Use a specific tokenizer or model.</li> <li>Use a <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> for audio, vision, and multimodal tasks.</li></ul> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-18z8fsr">Take a look at the <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> documentation for a complete list of supported tasks and available parameters.</p></div> <h2 class="relative group"><a id="pipeline-usage" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#pipeline-usage"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Pipeline usage</span></h2> <p data-svelte-h="svelte-1q7rbqu">While each task has an associated <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a>, it is simpler to use the general <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> abstraction which contains | |
| all the task-specific pipelines. The <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> automatically loads a default model and a preprocessing class capable | |
| of inference for your task. Let’s take the example of using the <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> for automatic speech recognition (ASR), or | |
| speech-to-text.</p> <ol data-svelte-h="svelte-ffga0r"><li>Start by creating a <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> and specify the inference task:</li></ol> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| <span class="hljs-meta">>>> </span>transcriber = pipeline(task=<span class="hljs-string">"automatic-speech-recognition"</span>)<!-- HTML_TAG_END --></pre></div> <ol start="2" data-svelte-h="svelte-10juiuh"><li>Pass your input to the <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a>. In the case of speech recognition, this is an audio input file:</li></ol> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>transcriber(<span class="hljs-string">"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac"</span>) | |
| {<span class="hljs-string">'text'</span>: <span class="hljs-string">'I HAVE A DREAM BUT ONE DAY THIS NATION WILL RISE UP LIVE UP THE TRUE MEANING OF ITS TREES'</span>}<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-jt3v05">Not the result you had in mind? Check out some of the <a href="https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&sort=trending" rel="nofollow">most downloaded automatic speech recognition models</a> | |
| on the Hub to see if you can get a better transcription.</p> <p data-svelte-h="svelte-1buznkw">Let’s try the <a href="https://huggingface.co/openai/whisper-large-v2" rel="nofollow">Whisper large-v2</a> model from OpenAI. Whisper was released | |
| 2 years later than Wav2Vec2, and was trained on close to 10x more data. As such, it beats Wav2Vec2 on most downstream | |
| benchmarks. It also has the added benefit of predicting punctuation and casing, neither of which are possible with<br> | |
| Wav2Vec2.</p> <p data-svelte-h="svelte-5gg07s">Let’s give it a try here to see how it performs:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>transcriber = pipeline(model=<span class="hljs-string">"openai/whisper-large-v2"</span>) | |
| <span class="hljs-meta">>>> </span>transcriber(<span class="hljs-string">"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac"</span>) | |
| {<span class="hljs-string">'text'</span>: <span class="hljs-string">' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'</span>}<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1yjypga">Now this result looks more accurate! For a deep-dive comparison on Wav2Vec2 vs Whisper, refer to the <a href="https://huggingface.co/learn/audio-course/chapter5/asr_models" rel="nofollow">Audio Transformers Course</a>. | |
| We really encourage you to check out the Hub for models in different languages, models specialized in your field, and more. | |
| You can check out and compare model results directly from your browser on the Hub to see if it fits or | |
| handles corner cases better than other ones. | |
| And if you don’t find a model for your use case, you can always start <a href="training">training</a> your own!</p> <p data-svelte-h="svelte-1udq0dv">If you have several inputs, you can pass your input as a list:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->transcriber( | |
| [ | |
| <span class="hljs-string">"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac"</span>, | |
| <span class="hljs-string">"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac"</span>, | |
| ] | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1y2bxsh">Pipelines are great for experimentation as switching from one model to another is trivial; however, there are some ways to optimize them for larger workloads than experimentation. See the following guides that dive into iterating over whole datasets or using pipelines in a webserver: | |
| of the docs:</p> <ul data-svelte-h="svelte-boqb6q"><li><a href="#using-pipelines-on-a-dataset">Using pipelines on a dataset</a></li> <li><a href="./pipeline_webserver">Using pipelines for a webserver</a></li></ul> <h2 class="relative group"><a id="parameters" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#parameters"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Parameters</span></h2> <p data-svelte-h="svelte-16x1qfq"><a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> supports many parameters; some are task specific, and some are general to all pipelines. | |
| In general, you can specify parameters anywhere you want:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->transcriber = pipeline(model=<span class="hljs-string">"openai/whisper-large-v2"</span>, my_parameter=<span class="hljs-number">1</span>) | |
| out = transcriber(...) <span class="hljs-comment"># This will use `my_parameter=1`.</span> | |
| out = transcriber(..., my_parameter=<span class="hljs-number">2</span>) <span class="hljs-comment"># This will override and use `my_parameter=2`.</span> | |
| out = transcriber(...) <span class="hljs-comment"># This will go back to using `my_parameter=1`.</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-e9485z">Let’s check out 3 important ones:</p> <h3 class="relative group"><a id="device" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#device"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Device</span></h3> <p data-svelte-h="svelte-sob7ne">If you use <code>device=n</code>, the pipeline automatically puts the model on the specified device. | |
| This will work regardless of whether you are using PyTorch or Tensorflow.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->transcriber = pipeline(model=<span class="hljs-string">"openai/whisper-large-v2"</span>, device=<span class="hljs-number">0</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1lln0af">If the model is too large for a single GPU and you are using PyTorch, you can set <code>torch_dtype='float16'</code> to enable FP16 precision inference. Usually this would not cause significant performance drops but make sure you evaluate it on your models!</p> <p data-svelte-h="svelte-197qrar">Alternatively, you can set <code>device_map="auto"</code> to automatically | |
| determine how to load and store the model weights. Using the <code>device_map</code> argument requires the 🤗 <a href="https://huggingface.co/docs/accelerate" rel="nofollow">Accelerate</a> | |
| package:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->pip install --upgrade accelerate<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-6riuku">The following code automatically loads and stores model weights across devices:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->transcriber = pipeline(model=<span class="hljs-string">"openai/whisper-large-v2"</span>, device_map=<span class="hljs-string">"auto"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1otna9r">Note that if <code>device_map="auto"</code> is passed, there is no need to add the argument <code>device=device</code> when instantiating your <code>pipeline</code> as you may encounter some unexpected behavior!</p> <h3 class="relative group"><a id="batch-size" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#batch-size"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Batch size</span></h3> <p data-svelte-h="svelte-v1mumi">By default, pipelines will not batch inference for reasons explained in detail <a href="https://huggingface.co/docs/transformers/main_classes/pipelines#pipeline-batching" rel="nofollow">here</a>. The reason is that batching is not necessarily faster, and can actually be quite slower in some cases.</p> <p data-svelte-h="svelte-1jutax8">But if it works in your use case, you can use:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->transcriber = pipeline(model=<span class="hljs-string">"openai/whisper-large-v2"</span>, device=<span class="hljs-number">0</span>, batch_size=<span class="hljs-number">2</span>) | |
| audio_filenames = [<span class="hljs-string">f"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/<span class="hljs-subst">{i}</span>.flac"</span> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">1</span>, <span class="hljs-number">5</span>)] | |
| texts = transcriber(audio_filenames)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-n3ouik">This runs the pipeline on the 4 provided audio files, but it will pass them in batches of 2 | |
| to the model (which is on a GPU, where batching is more likely to help) without requiring any further code from you. | |
| The output should always match what you would have received without batching. It is only meant as a way to help you get more speed out of a pipeline.</p> <p data-svelte-h="svelte-sq1wry">Pipelines can also alleviate some of the complexities of batching because, for some pipelines, a single item (like a long audio file) needs to be chunked into multiple parts to be processed by a model. The pipeline performs this <a href="./main_classes/pipelines#pipeline-chunk-batching"><em>chunk batching</em></a> for you.</p> <h3 class="relative group"><a id="task-specific-parameters" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#task-specific-parameters"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Task specific parameters</span></h3> <p data-svelte-h="svelte-16kv285">All tasks provide task specific parameters which allow for additional flexibility and options to help you get your job done. | |
| For instance, the <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline.__call__">transformers.AutomaticSpeechRecognitionPipeline.<strong>call</strong>()</a> method has a <code>return_timestamps</code> parameter which sounds promising for subtitling videos:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>transcriber = pipeline(model=<span class="hljs-string">"openai/whisper-large-v2"</span>, return_timestamps=<span class="hljs-literal">True</span>) | |
| <span class="hljs-meta">>>> </span>transcriber(<span class="hljs-string">"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac"</span>) | |
| {<span class="hljs-string">'text'</span>: <span class="hljs-string">' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'</span>, <span class="hljs-string">'chunks'</span>: [{<span class="hljs-string">'timestamp'</span>: (<span class="hljs-number">0.0</span>, <span class="hljs-number">11.88</span>), <span class="hljs-string">'text'</span>: <span class="hljs-string">' I have a dream that one day this nation will rise up and live out the true meaning of its'</span>}, {<span class="hljs-string">'timestamp'</span>: (<span class="hljs-number">11.88</span>, <span class="hljs-number">12.38</span>), <span class="hljs-string">'text'</span>: <span class="hljs-string">' creed.'</span>}]}<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1ad1wem">As you can see, the model inferred the text and also outputted <strong>when</strong> the various sentences were pronounced.</p> <p data-svelte-h="svelte-1l2mis5">There are many parameters available for each task, so check out each task’s API reference to see what you can tinker with! | |
| For instance, the <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline">AutomaticSpeechRecognitionPipeline</a> has a <code>chunk_length_s</code> parameter which is helpful | |
| for working on really long audio files (for example, subtitling entire movies or hour-long videos) that a model typically | |
| cannot handle on its own:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span>transcriber = pipeline(model=<span class="hljs-string">"openai/whisper-large-v2"</span>, chunk_length_s=<span class="hljs-number">30</span>) | |
| <span class="hljs-meta">>>> </span>transcriber(<span class="hljs-string">"https://huggingface.co/datasets/reach-vb/random-audios/resolve/main/ted_60.wav"</span>) | |
| {<span class="hljs-string">'text'</span>: <span class="hljs-string">" So in college, I was a government major, which means I had to write a lot of papers. Now, when a normal student writes a paper, they might spread the work out a little like this. So, you know. You get started maybe a little slowly, but you get enough done in the first week that with some heavier days later on, everything gets done and things stay civil. And I would want to do that like that. That would be the plan. I would have it all ready to go, but then actually the paper would come along, and then I would kind of do this. And that would happen every single paper. But then came my 90-page senior thesis, a paper you're supposed to spend a year on. I knew for a paper like that, my normal workflow was not an option, it was way too big a project. So I planned things out and I decided I kind of had to go something like this. This is how the year would go. So I'd start off light and I'd bump it up"</span>}<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1nowgzq">If you can’t find a parameter that would really help you out, feel free to <a href="https://github.com/huggingface/transformers/issues/new?assignees=&labels=feature&template=feature-request.yml" rel="nofollow">request it</a>!</p> <h2 class="relative group"><a id="using-pipelines-on-a-dataset" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#using-pipelines-on-a-dataset"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Using pipelines on a dataset</span></h2> <p data-svelte-h="svelte-1sh6mg6">The pipeline can also run inference on a large dataset. The easiest way we recommend doing this is by using an iterator:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">def</span> <span class="hljs-title function_">data</span>(): | |
| <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">1000</span>): | |
| <span class="hljs-keyword">yield</span> <span class="hljs-string">f"My example <span class="hljs-subst">{i}</span>"</span> | |
| pipe = pipeline(model=<span class="hljs-string">"openai-community/gpt2"</span>, device=<span class="hljs-number">0</span>) | |
| generated_characters = <span class="hljs-number">0</span> | |
| <span class="hljs-keyword">for</span> out <span class="hljs-keyword">in</span> pipe(data()): | |
| generated_characters += <span class="hljs-built_in">len</span>(out[<span class="hljs-number">0</span>][<span class="hljs-string">"generated_text"</span>])<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1rrzg9">The iterator <code>data()</code> yields each result, and the pipeline automatically | |
| recognizes the input is iterable and will start fetching the data while | |
| it continues to process it on the GPU (this uses <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow">DataLoader</a> under the hood). | |
| This is important because you don’t have to allocate memory for the whole dataset | |
| and you can feed the GPU as fast as possible.</p> <p data-svelte-h="svelte-1litpj1">Since batching could speed things up, it may be useful to try tuning the <code>batch_size</code> parameter here.</p> <p data-svelte-h="svelte-1pgxdbs">The simplest way to iterate over a dataset is to just load one from 🤗 <a href="https://github.com/huggingface/datasets/" rel="nofollow">Datasets</a>:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># KeyDataset is a util that will just output the item we're interested in.</span> | |
| <span class="hljs-keyword">from</span> transformers.pipelines.pt_utils <span class="hljs-keyword">import</span> KeyDataset | |
| <span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| pipe = pipeline(model=<span class="hljs-string">"hf-internal-testing/tiny-random-wav2vec2"</span>, device=<span class="hljs-number">0</span>) | |
| dataset = load_dataset(<span class="hljs-string">"hf-internal-testing/librispeech_asr_dummy"</span>, <span class="hljs-string">"clean"</span>, split=<span class="hljs-string">"validation[:10]"</span>) | |
| <span class="hljs-keyword">for</span> out <span class="hljs-keyword">in</span> pipe(KeyDataset(dataset, <span class="hljs-string">"audio"</span>)): | |
| <span class="hljs-built_in">print</span>(out)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="using-pipelines-for-a-webserver" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#using-pipelines-for-a-webserver"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Using pipelines for a webserver</span></h2> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400">Creating an inference engine is a complex topic which deserves it's own | |
| page.</div> <p data-svelte-h="svelte-52dmam"><a href="./pipeline_webserver">Link</a></p> <h2 class="relative group"><a id="vision-pipeline" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#vision-pipeline"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Vision pipeline</span></h2> <p data-svelte-h="svelte-3x0539">Using a <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> for vision tasks is practically identical.</p> <p data-svelte-h="svelte-ip2z2f">Specify your task and pass your image to the classifier. The image can be a link, a local path or a base64-encoded image. For example, what species of cat is shown below?</p> <p data-svelte-h="svelte-1bks2jw"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" alt="pipeline-cat-chonk"></p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| <span class="hljs-meta">>>> </span>vision_classifier = pipeline(model=<span class="hljs-string">"google/vit-base-patch16-224"</span>) | |
| <span class="hljs-meta">>>> </span>preds = vision_classifier( | |
| <span class="hljs-meta">... </span> images=<span class="hljs-string">"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"</span> | |
| <span class="hljs-meta">... </span>) | |
| <span class="hljs-meta">>>> </span>preds = [{<span class="hljs-string">"score"</span>: <span class="hljs-built_in">round</span>(pred[<span class="hljs-string">"score"</span>], <span class="hljs-number">4</span>), <span class="hljs-string">"label"</span>: pred[<span class="hljs-string">"label"</span>]} <span class="hljs-keyword">for</span> pred <span class="hljs-keyword">in</span> preds] | |
| <span class="hljs-meta">>>> </span>preds | |
| [{<span class="hljs-string">'score'</span>: <span class="hljs-number">0.4335</span>, <span class="hljs-string">'label'</span>: <span class="hljs-string">'lynx, catamount'</span>}, {<span class="hljs-string">'score'</span>: <span class="hljs-number">0.0348</span>, <span class="hljs-string">'label'</span>: <span class="hljs-string">'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'</span>}, {<span class="hljs-string">'score'</span>: <span class="hljs-number">0.0324</span>, <span class="hljs-string">'label'</span>: <span class="hljs-string">'snow leopard, ounce, Panthera uncia'</span>}, {<span class="hljs-string">'score'</span>: <span class="hljs-number">0.0239</span>, <span class="hljs-string">'label'</span>: <span class="hljs-string">'Egyptian cat'</span>}, {<span class="hljs-string">'score'</span>: <span class="hljs-number">0.0229</span>, <span class="hljs-string">'label'</span>: <span class="hljs-string">'tiger cat'</span>}]<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="text-pipeline" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#text-pipeline"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Text pipeline</span></h2> <p data-svelte-h="svelte-1a24e0d">Using a <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> for NLP tasks is practically identical.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># This model is a `zero-shot-classification` model.</span> | |
| <span class="hljs-meta">>>> </span><span class="hljs-comment"># It will classify text, except you are free to choose any label you might imagine</span> | |
| <span class="hljs-meta">>>> </span>classifier = pipeline(model=<span class="hljs-string">"facebook/bart-large-mnli"</span>) | |
| <span class="hljs-meta">>>> </span>classifier( | |
| <span class="hljs-meta">... </span> <span class="hljs-string">"I have a problem with my iphone that needs to be resolved asap!!"</span>, | |
| <span class="hljs-meta">... </span> candidate_labels=[<span class="hljs-string">"urgent"</span>, <span class="hljs-string">"not urgent"</span>, <span class="hljs-string">"phone"</span>, <span class="hljs-string">"tablet"</span>, <span class="hljs-string">"computer"</span>], | |
| <span class="hljs-meta">... </span>) | |
| {<span class="hljs-string">'sequence'</span>: <span class="hljs-string">'I have a problem with my iphone that needs to be resolved asap!!'</span>, <span class="hljs-string">'labels'</span>: [<span class="hljs-string">'urgent'</span>, <span class="hljs-string">'phone'</span>, <span class="hljs-string">'computer'</span>, <span class="hljs-string">'not urgent'</span>, <span class="hljs-string">'tablet'</span>], <span class="hljs-string">'scores'</span>: [<span class="hljs-number">0.504</span>, <span class="hljs-number">0.479</span>, <span class="hljs-number">0.013</span>, <span class="hljs-number">0.003</span>, <span class="hljs-number">0.002</span>]}<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="multimodal-pipeline" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#multimodal-pipeline"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Multimodal pipeline</span></h2> <p data-svelte-h="svelte-10ds082">The <a href="/docs/transformers/pr_33913/en/main_classes/pipelines#transformers.pipeline">pipeline()</a> supports more than one modality. For example, a visual question answering (VQA) task combines text and image. Feel free to use any image link you like and a question you want to ask about the image. The image can be a URL or a local path to the image.</p> <p data-svelte-h="svelte-4s0csr">For example, if you use this <a href="https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/invoice.png" rel="nofollow">invoice image</a>:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| <span class="hljs-meta">>>> </span>vqa = pipeline(model=<span class="hljs-string">"impira/layoutlm-document-qa"</span>) | |
| <span class="hljs-meta">>>> </span>output = vqa( | |
| <span class="hljs-meta">... </span> image=<span class="hljs-string">"https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/invoice.png"</span>, | |
| <span class="hljs-meta">... </span> question=<span class="hljs-string">"What is the invoice number?"</span>, | |
| <span class="hljs-meta">... </span>) | |
| <span class="hljs-meta">>>> </span>output[<span class="hljs-number">0</span>][<span class="hljs-string">"score"</span>] = <span class="hljs-built_in">round</span>(output[<span class="hljs-number">0</span>][<span class="hljs-string">"score"</span>], <span class="hljs-number">3</span>) | |
| <span class="hljs-meta">>>> </span>output | |
| [{<span class="hljs-string">'score'</span>: <span class="hljs-number">0.425</span>, <span class="hljs-string">'answer'</span>: <span class="hljs-string">'us-001'</span>, <span class="hljs-string">'start'</span>: <span class="hljs-number">16</span>, <span class="hljs-string">'end'</span>: <span class="hljs-number">16</span>}]<!-- HTML_TAG_END --></pre></div> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-z3k1tk">To run the example above you need to have <a href="https://pypi.org/project/pytesseract/" rel="nofollow"><code>pytesseract</code></a> installed in addition to 🤗 Transformers:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->sudo apt install -y tesseract-ocr | |
| pip install pytesseract<!-- HTML_TAG_END --></pre></div></div> <h2 class="relative group"><a id="using-pipeline-on-large-models-with--accelerate-" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#using-pipeline-on-large-models-with--accelerate-"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Using pipeline on large models with 🤗 accelerate :</span></h2> <p data-svelte-h="svelte-32c7q2">You can easily run <code>pipeline</code> on large models using 🤗 <code>accelerate</code>! First make sure you have installed <code>accelerate</code> with <code>pip install accelerate</code>.</p> <p data-svelte-h="svelte-1yg483v">First load your model using <code>device_map="auto"</code>! We will use <code>facebook/opt-1.3b</code> for our example.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># pip install accelerate</span> | |
| <span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| pipe = pipeline(model=<span class="hljs-string">"facebook/opt-1.3b"</span>, torch_dtype=torch.bfloat16, device_map=<span class="hljs-string">"auto"</span>) | |
| output = pipe(<span class="hljs-string">"This is a cool example!"</span>, do_sample=<span class="hljs-literal">True</span>, top_p=<span class="hljs-number">0.95</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1c9ycn6">You can also pass 8-bit loaded models if you install <code>bitsandbytes</code> and add the argument <code>load_in_8bit=True</code></p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># pip install accelerate bitsandbytes</span> | |
| <span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| pipe = pipeline(model=<span class="hljs-string">"facebook/opt-1.3b"</span>, device_map=<span class="hljs-string">"auto"</span>, model_kwargs={<span class="hljs-string">"load_in_8bit"</span>: <span class="hljs-literal">True</span>}) | |
| output = pipe(<span class="hljs-string">"This is a cool example!"</span>, do_sample=<span class="hljs-literal">True</span>, top_p=<span class="hljs-number">0.95</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-frv3wv">Note that you can replace the checkpoint with any Hugging Face model that supports large model loading, such as BLOOM.</p> <h2 class="relative group"><a id="creating-web-demos-from-pipelines-with-gradio" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#creating-web-demos-from-pipelines-with-gradio"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Creating web demos from pipelines with gradio</span></h2> <p data-svelte-h="svelte-hf8ov5">Pipelines are automatically supported in <a href="https://github.com/gradio-app/gradio/" rel="nofollow">Gradio</a>, a library that makes creating beautiful and user-friendly machine learning apps on the web a breeze. First, make sure you have Gradio installed:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->pip <span class="hljs-keyword">install</span> gradio<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-191d3mo">Then, you can create a web demo around an image classification pipeline (or any other pipeline) in a single line of code by calling Gradio’s <a href="https://www.gradio.app/docs/interface#interface-from-pipeline" rel="nofollow"><code>Interface.from_pipeline</code></a> function to launch the pipeline. This creates an intuitive drag-and-drop interface in your browser:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| <span class="hljs-keyword">import</span> gradio <span class="hljs-keyword">as</span> gr | |
| pipe = pipeline(<span class="hljs-string">"image-classification"</span>, model=<span class="hljs-string">"google/vit-base-patch16-224"</span>) | |
| gr.Interface.from_pipeline(pipe).launch()<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-rjj5jb"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/panda-classification.png"></p> <p data-svelte-h="svelte-efanlv">By default, the web demo runs on a local server. If you’d like to share it with others, you can generate a temporary public | |
| link by setting <code>share=True</code> in <code>launch()</code>. You can also host your demo on <a href="https://huggingface.co/spaces" rel="nofollow">Hugging Face Spaces</a> for a permanent link.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_z647wz = { | |
| assets: "/docs/transformers/pr_33913/en", | |
| base: "/docs/transformers/pr_33913/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_33913/en/_app/immutable/entry/start.b67f883f.js"), | |
| import("/docs/transformers/pr_33913/en/_app/immutable/entry/app.e436b1f2.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 392], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 75.4 kB
- Xet hash:
- 9e7197cb898dec1e60b6ec6da9d6323e4fec1e5380599d1affb8fe1fb1030516
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.