Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Web server inference","local":"web-server-inference","sections":[{"title":"Create a server","local":"create-a-server","sections":[],"depth":2},{"title":"Queuing requests","local":"queuing-requests","sections":[],"depth":2},{"title":"Error checking","local":"error-checking","sections":[],"depth":2},{"title":"Circuit breaking","local":"circuit-breaking","sections":[],"depth":2},{"title":"Block the main thread","local":"block-the-main-thread","sections":[],"depth":2},{"title":"Dynamic batching","local":"dynamic-batching","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_36839/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/entry/start.6be8d590.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/scheduler.01eeda35.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/singletons.177df05e.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/index.4862150a.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/paths.517376d1.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/entry/app.09748b4b.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/index.6dd51b66.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/nodes/0.8897c14d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/nodes/428.e46e82a8.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/CodeBlock.864da1b0.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/EditOnGithub.7faefd25.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Web server inference","local":"web-server-inference","sections":[{"title":"Create a server","local":"create-a-server","sections":[],"depth":2},{"title":"Queuing requests","local":"queuing-requests","sections":[],"depth":2},{"title":"Error checking","local":"error-checking","sections":[],"depth":2},{"title":"Circuit breaking","local":"circuit-breaking","sections":[],"depth":2},{"title":"Block the main thread","local":"block-the-main-thread","sections":[],"depth":2},{"title":"Dynamic batching","local":"dynamic-batching","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="web-server-inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#web-server-inference"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Web server inference</span></h1> <p data-svelte-h="svelte-1von711">A web server is a system that waits for requests and serves them as they come in. This means you can use <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.Pipeline">Pipeline</a> as an inference engine on a web server, since you can use an iterator (similar to how you would <a href="./pipeline_tutorial#large-datasets">iterate over a dataset</a>) to handle each incoming request.</p> <p data-svelte-h="svelte-z14khm">Designing a web server with <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.Pipeline">Pipeline</a> is unique though because they’re fundamentally different. Web servers are multiplexed (multithreaded, async, etc.) to handle multiple requests concurrently. <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.Pipeline">Pipeline</a> and its underlying model on the other hand are not designed for parallelism because they take a lot of memory. It’s best to give a <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.Pipeline">Pipeline</a> all the available resources when they’re running or for a compute intensive job.</p> <p data-svelte-h="svelte-1lr7px0">This guide shows how to work around this difference by using a web server to handle the lighter load of receiving and sending requests, and having a single thread to handle the heavier load of running <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.Pipeline">Pipeline</a>.</p> <h2 class="relative group"><a id="create-a-server" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#create-a-server"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Create a server</span></h2> <p data-svelte-h="svelte-75xf7k"><a href="https://www.starlette.io/" rel="nofollow">Starlette</a> is a lightweight framework for building web servers. You can use any other framework you’d like, but you may have to make some changes to the code below.</p> <p data-svelte-h="svelte-vc63tw">Before you begin, make sure Starlette and <a href="http://www.uvicorn.org/" rel="nofollow">uvicorn</a> are installed.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->!pip install starlette uvicorn<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-hg1gr3">Now you can create a simple web server in a <code>server.py</code> file. The key is to only load the model <strong>once</strong> to prevent unnecessary copies of it from consuming memory.</p> <p data-svelte-h="svelte-1rmfx7s">Create a pipeline to fill in the masked token, <code>[MASK]</code>.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> starlette.applications <span class="hljs-keyword">import</span> Starlette | |
| <span class="hljs-keyword">from</span> starlette.responses <span class="hljs-keyword">import</span> JSONResponse | |
| <span class="hljs-keyword">from</span> starlette.routing <span class="hljs-keyword">import</span> Route | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| <span class="hljs-keyword">import</span> asyncio | |
| <span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">homepage</span>(<span class="hljs-params">request</span>): | |
| payload = <span class="hljs-keyword">await</span> request.body() | |
| string = payload.decode(<span class="hljs-string">"utf-8"</span>) | |
| response_q = asyncio.Queue() | |
| <span class="hljs-keyword">await</span> request.app.model_queue.put((string, response_q)) | |
| output = <span class="hljs-keyword">await</span> response_q.get() | |
| <span class="hljs-keyword">return</span> JSONResponse(output) | |
| <span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">server_loop</span>(<span class="hljs-params">q</span>): | |
| pipeline = pipeline(task=<span class="hljs-string">"fill-mask"</span>,model=<span class="hljs-string">"google-bert/bert-base-uncased"</span>) | |
| <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>: | |
| (string, response_q) = <span class="hljs-keyword">await</span> q.get() | |
| out = pipeline(string) | |
| <span class="hljs-keyword">await</span> response_q.put(out) | |
| app = Starlette( | |
| routes=[ | |
| Route(<span class="hljs-string">"/"</span>, homepage, methods=[<span class="hljs-string">"POST"</span>]), | |
| ], | |
| ) | |
| <span class="hljs-meta">@app.on_event(<span class="hljs-params"><span class="hljs-string">"startup"</span></span>)</span> | |
| <span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">startup_event</span>(): | |
| q = asyncio.Queue() | |
| app.model_queue = q | |
| asyncio.create_task(server_loop(q))<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-khywk9">Start the server with the following command.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->uvicorn server:app<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-wy3f7i">Query the server with a POST request.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->curl -X POST -d <span class="hljs-string">"Paris is the [MASK] of France."</span> http://localhost:8000/ | |
| [{<span class="hljs-string">'score'</span>: 0.9969332218170166, | |
| <span class="hljs-string">'token'</span>: 3007, | |
| <span class="hljs-string">'token_str'</span>: <span class="hljs-string">'capital'</span>, | |
| <span class="hljs-string">'sequence'</span>: <span class="hljs-string">'paris is the capital of france.'</span>}, | |
| {<span class="hljs-string">'score'</span>: 0.0005914849461987615, | |
| <span class="hljs-string">'token'</span>: 2540, | |
| <span class="hljs-string">'token_str'</span>: <span class="hljs-string">'heart'</span>, | |
| <span class="hljs-string">'sequence'</span>: <span class="hljs-string">'paris is the heart of france.'</span>}, | |
| {<span class="hljs-string">'score'</span>: 0.00043787318281829357, | |
| <span class="hljs-string">'token'</span>: 2415, | |
| <span class="hljs-string">'token_str'</span>: <span class="hljs-string">'center'</span>, | |
| <span class="hljs-string">'sequence'</span>: <span class="hljs-string">'paris is the center of france.'</span>}, | |
| {<span class="hljs-string">'score'</span>: 0.0003378340043127537, | |
| <span class="hljs-string">'token'</span>: 2803, | |
| <span class="hljs-string">'token_str'</span>: <span class="hljs-string">'centre'</span>, | |
| <span class="hljs-string">'sequence'</span>: <span class="hljs-string">'paris is the centre of france.'</span>}, | |
| {<span class="hljs-string">'score'</span>: 0.00026995912776328623, | |
| <span class="hljs-string">'token'</span>: 2103, | |
| <span class="hljs-string">'token_str'</span>: <span class="hljs-string">'city'</span>, | |
| <span class="hljs-string">'sequence'</span>: <span class="hljs-string">'paris is the city of france.'</span>}]<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="queuing-requests" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#queuing-requests"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Queuing requests</span></h2> <p data-svelte-h="svelte-1pq1hmu">The server’s queuing mechanism can be used for some interesting applications such as dynamic batching. Dynamic batching accumulates several requests first before processing them with <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.Pipeline">Pipeline</a>.</p> <p data-svelte-h="svelte-o8k0wa">The example below is written in pseudocode for readability rather than performance, in particular, you’ll notice that:</p> <ol data-svelte-h="svelte-12z6vc1"><li><p>There is no batch size limit.</p></li> <li><p>The timeout is reset on every queue fetch, so you could end up waiting much longer than the <code>timeout</code> value before processing a request. This would also delay the first inference request by that amount of time. The web server always waits 1ms even if the queue is empty, which is inefficient, because that time can be used to start inference. It could make sense though if batching is essential to your use case.</p> <p>It would be better to have a single 1ms deadline, instead of resetting it on every fetch.</p></li></ol> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->(string, rq) = <span class="hljs-keyword">await</span> q.get() | |
| strings = [] | |
| queues = [] | |
| <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>: | |
| <span class="hljs-keyword">try</span>: | |
| (string, rq) = <span class="hljs-keyword">await</span> asyncio.wait_for(q.get(), timeout=<span class="hljs-number">0.001</span>) | |
| <span class="hljs-keyword">except</span> asyncio.exceptions.TimeoutError: | |
| <span class="hljs-keyword">break</span> | |
| strings.append(string) | |
| queues.append(rq) | |
| strings | |
| outs = pipeline(strings, batch_size=<span class="hljs-built_in">len</span>(strings)) | |
| <span class="hljs-keyword">for</span> rq, out <span class="hljs-keyword">in</span> <span class="hljs-built_in">zip</span>(queues, outs): | |
| <span class="hljs-keyword">await</span> rq.put(out)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="error-checking" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#error-checking"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Error checking</span></h2> <p data-svelte-h="svelte-np02fr">There are many things that can go wrong in production. You could run out-of-memory, out of space, fail to load a model, have an incorrect model configuration, have an incorrect query, and so much more.</p> <p data-svelte-h="svelte-18bu5an">Adding <code>try...except</code> statements is helpful for returning these errors to the user for debugging. Keep in mind this could be a security risk if you shouldn’t be revealing certain information.</p> <h2 class="relative group"><a id="circuit-breaking" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#circuit-breaking"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Circuit breaking</span></h2> <p data-svelte-h="svelte-q5t78j">Try to return a 503 or 504 error when the server is overloaded instead of forcing a user to wait indefinitely.</p> <p data-svelte-h="svelte-b9g89k">It is relatively simple to implement these error types since it’s only a single queue. Take a look at the queue size to determine when to start returning errors before your server fails under load.</p> <h2 class="relative group"><a id="block-the-main-thread" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#block-the-main-thread"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Block the main thread</span></h2> <p data-svelte-h="svelte-vk268r">PyTorch is not async aware, so computation will block the main thread from running.</p> <p data-svelte-h="svelte-1am6qf5">For this reason, it’s better to run PyTorch on its own separate thread or process. When inference of a single request is especially long (more than 1s), it’s even more important because it means every query during inference must wait 1s before even receiving an error.</p> <h2 class="relative group"><a id="dynamic-batching" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#dynamic-batching"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Dynamic batching</span></h2> <p data-svelte-h="svelte-19c4yu0">Dynamic batching can be very effective when used in the correct setting, but it’s not necessary when you’re only passing 1 request at a time (see <a href="./pipeline_tutorial#batch-inference">batch inference</a> for more details).</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_webserver.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_1bm5psi = { | |
| assets: "/docs/transformers/pr_36839/en", | |
| base: "/docs/transformers/pr_36839/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_36839/en/_app/immutable/entry/start.6be8d590.js"), | |
| import("/docs/transformers/pr_36839/en/_app/immutable/entry/app.09748b4b.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 428], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 28.4 kB
- Xet hash:
- b38b46410a4933981a54253bcff369d872ead2e21663e1f8c7a3907fddce3124
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.