Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Create custom Inference Handler","local":"create-custom-inference-handler","sections":[{"title":"Custom Handler Examples","local":"custom-handler-examples","sections":[],"depth":2},{"title":"Tutorial","local":"tutorial","sections":[{"title":"1. Set up Development Environment","local":"1-set-up-development-environment","sections":[],"depth":3},{"title":"2. Create EndpointHandler","local":"2-create-endpointhandler","sections":[],"depth":3},{"title":"3. Customize EndpointHandler","local":"3-customize-endpointhandler","sections":[],"depth":3},{"title":"4. Test EndpointHandler","local":"4-test-endpointhandler","sections":[],"depth":3},{"title":"5. Push the Custom Handler to your repository","local":"5-push-the-custom-handler-to-your-repository","sections":[],"depth":3},{"title":"6. Deploy your Custom Handler as an Inference Endpoint","local":"6-deploy-your-custom-handler-as-an-inference-endpoint","sections":[],"depth":3}],"depth":2}],"depth":1}"> | |
| <link href="/docs/inference-endpoints/pr_113/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/entry/start.d1c14968.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/chunks/scheduler.389d799c.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/chunks/singletons.16c9b508.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/chunks/paths.58d119e0.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/entry/app.18050d92.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/chunks/index.8f81d18f.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/nodes/0.ce016c16.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/nodes/11.b9de5b0f.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/chunks/CodeBlock.c0898180.js"> | |
| <link rel="modulepreload" href="/docs/inference-endpoints/pr_113/en/_app/immutable/chunks/getInferenceSnippets.8efa8e08.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Create custom Inference Handler","local":"create-custom-inference-handler","sections":[{"title":"Custom Handler Examples","local":"custom-handler-examples","sections":[],"depth":2},{"title":"Tutorial","local":"tutorial","sections":[{"title":"1. Set up Development Environment","local":"1-set-up-development-environment","sections":[],"depth":3},{"title":"2. Create EndpointHandler","local":"2-create-endpointhandler","sections":[],"depth":3},{"title":"3. Customize EndpointHandler","local":"3-customize-endpointhandler","sections":[],"depth":3},{"title":"4. Test EndpointHandler","local":"4-test-endpointhandler","sections":[],"depth":3},{"title":"5. Push the Custom Handler to your repository","local":"5-push-the-custom-handler-to-your-repository","sections":[],"depth":3},{"title":"6. Deploy your Custom Handler as an Inference Endpoint","local":"6-deploy-your-custom-handler-as-an-inference-endpoint","sections":[],"depth":3}],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="create-custom-inference-handler" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#create-custom-inference-handler"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Create custom Inference Handler</span></h1> <p data-svelte-h="svelte-utxjdb">Hugging Face Endpoints supports all of the Transformers and Sentence-Transformers tasks and can support custom tasks, including custom pre- & post-processing. The customization can be done through a <a href="https://huggingface.co/philschmid/distilbert-onnx-banking77/blob/main/handler.py" rel="nofollow">handler.py</a> file in your model repository on the Hugging Face Hub.</p> <p data-svelte-h="svelte-nd5753">The <a href="https://huggingface.co/philschmid/distilbert-onnx-banking77/blob/main/handler.py" rel="nofollow">handler.py</a> needs to implement the <a href="https://huggingface.co/philschmid/distilbert-onnx-banking77/blob/main/handler.py" rel="nofollow">EndpointHandler</a> class with a <code>__init__</code> and a <code>__call__</code> method.</p> <p data-svelte-h="svelte-rirnor">If you want to use custom dependencies, e.g. <a href="https://raw.githubusercontent.com/huggingface/optimum" rel="nofollow">optimum</a>, the dependencies must be listed in a <code>requirements.txt</code> as described above in “add custom dependencies.”</p> <h2 class="relative group"><a id="custom-handler-examples" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#custom-handler-examples"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Custom Handler Examples</span></h2> <p data-svelte-h="svelte-6ei6y6">There are already several public examples on the <a href="https://huggingface.co/models?other=endpoints-template" rel="nofollow">Hugging Face Hub</a> where you can take insipiration or directly use them. The repositories are tagged with <code>endpoints-template</code> and can be found under this <a href="https://huggingface.co/models?other=endpoints-template" rel="nofollow">link</a>.</p> <p data-svelte-h="svelte-wxy1ws">Included examples are for:</p> <ul data-svelte-h="svelte-hgiuvk"><li><a href="https://huggingface.co/philschmid/distilbert-onnx-banking77" rel="nofollow">Optimum and ONNX Runtime</a></li> <li><a href="https://huggingface.co/florentgbelidji/blip_image_embeddings" rel="nofollow">Image Embeddings with BLIP</a></li> <li><a href="https://huggingface.co/philschmid/trocr-base-printed" rel="nofollow">TrOCR for OCR Detection</a></li> <li><a href="https://huggingface.co/philschmid/all-MiniLM-L6-v2-optimum-embeddings" rel="nofollow">Optimized Sentence Transformers with Optimum</a></li> <li><a href="https://huggingface.co/philschmid/pyannote-speaker-diarization-endpoint" rel="nofollow">Pyannote Speaker diarization</a></li> <li><a href="https://huggingface.co/philschmid/layoutlm-funsd" rel="nofollow">LayoutLM</a></li> <li><a href="https://huggingface.co/philschmid/flair-ner-english-ontonotes-large" rel="nofollow">Flair NER</a></li> <li><a href="https://huggingface.co/philschmid/gpt-j-6B-fp16-sharded" rel="nofollow">GPT-J 6B Single GPU</a></li> <li><a href="https://huggingface.co/philschmid/donut-base-finetuned-cord-v2" rel="nofollow">Donut Document understanding</a></li> <li><a href="https://huggingface.co/philschmid/setfit-ag-news-endpoint" rel="nofollow">SetFit classifier</a></li></ul> <h2 class="relative group"><a id="tutorial" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#tutorial"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Tutorial</span></h2> <p data-svelte-h="svelte-1lc7u48">Before creating a Custom Handler, you need a Hugging Face Model repository with your model weights and an Access Token with <em>WRITE</em> access to the repository. To find, create and manage Access Tokens, click <a href="https://huggingface.co/settings/tokens" rel="nofollow">here</a>.</p> <p data-svelte-h="svelte-131h02t">If you want to write a Custom Handler for an existing model from the community, you can use the <a href="https://huggingface.co/spaces/osanseviero/repo_duplicator" rel="nofollow">repo_duplicator</a> to create a repository fork.</p> <p data-svelte-h="svelte-xuocio">The code can also be found in this <a href="https://colab.research.google.com/drive/1hANJeRa1PK1gZaUorobnQGu4bFj4_4Rf?usp=sharing" rel="nofollow">Notebook</a>.</p> <p data-svelte-h="svelte-16in619">You can also search for already existing Custom Handlers here: <a href="https://huggingface.co/models?other=endpoints-template" rel="nofollow">https://huggingface.co/models?other=endpoints-template</a></p> <h3 class="relative group"><a id="1-set-up-development-environment" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#1-set-up-development-environment"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>1. Set up Development Environment</span></h3> <p data-svelte-h="svelte-s875rd">The easiest way to develop our custom handler is to set up a local development environment, to implement, test, and iterate there, and then deploy it as an Inference Endpoint. The first step is to install all required development dependencies. | |
| <em>needed to create the custom handler, not needed for inference</em></p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># install git-lfs to interact with the repository</span> | |
| sudo apt-<span class="hljs-built_in">get</span> update | |
| sudo apt-<span class="hljs-built_in">get</span> install git-lfs | |
| <span class="hljs-comment"># install transformers (not needed since it is installed by default in the container)</span> | |
| pip install transformers[sklearn,sentencepiece,audio,vision]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-12klzx5">After we have installed our libraries we will clone our repository to our development environment.</p> <p data-svelte-h="svelte-ms7r4w">We will use <a href="https://huggingface.co/philschmid/distilbert-base-uncased-emotion" rel="nofollow">philschmid/distilbert-base-uncased-emotion</a> during the tutorial.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->git lfs install | |
| git clone https:<span class="hljs-regexp">//</span>huggingface.co<span class="hljs-regexp">/philschmid/</span>distilbert-base-uncased-emotion<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-54lmtl">To be able to push our model repo later you need to login into our HF account. This can be done by using the <code>huggingface-cli</code>.</p> <p data-svelte-h="svelte-k22wa5"><em>Note: Make sure to configure git config as well.</em></p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --># setup <span class="hljs-keyword">cli</span> with <span class="hljs-keyword">token</span> | |
| huggingface-<span class="hljs-keyword">cli</span> login | |
| git config --<span class="hljs-keyword">global</span> credential.helper store<!-- HTML_TAG_END --></pre></div> <h3 class="relative group"><a id="2-create-endpointhandler" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#2-create-endpointhandler"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>2. Create EndpointHandler</span></h3> <p data-svelte-h="svelte-1dqini1">After we have set up our environment, we can start creating your custom handler. The custom handler is a Python class (<code>EndpointHandler</code>) inside a <code>handler.py</code> file in our repository. The <code>EndpointHandler</code> needs to implement an <code>__init__</code> and a <code>__call__</code> method.</p> <ul data-svelte-h="svelte-1gdza5u"><li>The <code>__init__</code> method will be called when starting the Endpoint and will receive 1 argument, a string with the path to your model weights. This allows you to load your model correctly.</li> <li>The <code>__call__</code> method will be called on every request and receive a dictionary with your request body as a python dictionary. It will always contain the <code>inputs</code> key.</li></ul> <p data-svelte-h="svelte-1xwwutb">The first step is to create our <code>handler.py</code> in the local clone of our repository.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->!<span class="hljs-built_in">cd</span> distilbert-base-uncased-emotion && <span class="hljs-built_in">touch</span> handler.py<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-16cpa65">In there, you define your <code>EndpointHandler</code> class with the <code>__init__</code> and <code>__call__ </code>method.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> typing <span class="hljs-keyword">import</span> <span class="hljs-type">Dict</span>, <span class="hljs-type">List</span>, <span class="hljs-type">Any</span> | |
| <span class="hljs-keyword">class</span> <span class="hljs-title class_">EndpointHandler</span>(): | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, path=<span class="hljs-string">""</span></span>): | |
| <span class="hljs-comment"># Preload all the elements you are going to need at inference.</span> | |
| <span class="hljs-comment"># pseudo:</span> | |
| <span class="hljs-comment"># self.model= load_model(path)</span> | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">__call__</span>(<span class="hljs-params">self, data: <span class="hljs-type">Dict</span>[<span class="hljs-built_in">str</span>, <span class="hljs-type">Any</span>]</span>) -> <span class="hljs-type">List</span>[<span class="hljs-type">Dict</span>[<span class="hljs-built_in">str</span>, <span class="hljs-type">Any</span>]]: | |
| <span class="hljs-string">""" | |
| data args: | |
| inputs (:obj: `str` | `PIL.Image` | `np.array`) | |
| kwargs | |
| Return: | |
| A :obj:`list` | `dict`: will be serialized and returned | |
| """</span> | |
| <span class="hljs-comment"># pseudo</span> | |
| <span class="hljs-comment"># self.model(input)</span><!-- HTML_TAG_END --></pre></div> <h3 class="relative group"><a id="3-customize-endpointhandler" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#3-customize-endpointhandler"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>3. Customize EndpointHandler</span></h3> <p data-svelte-h="svelte-uax0tq">Now, you can add all of the custom logic you want to use during initialization or inference to your Custom Endpoint. You can already find multiple <a href="https://huggingface.co/models?other=endpoints-template" rel="nofollow">Custom Handler on the Hub</a> if you need some inspiration. In our example, we will add a custom condition based on additional payload information.</p> <p data-svelte-h="svelte-1f0seln"><em>The model we are using in the tutorial is fine-tuned to detect emotions. We will add an additional payload field for the date, and will use an external package to check if it is a holiday, to add a condition so that when the input date is a holiday, the model returns “happy” - since everyone is happy when there are holidays </em>🌴🎉😆</p> <p data-svelte-h="svelte-1fqhyg1">First, we need to create a new <code>requirements.txt</code> and add our <a href="https://pypi.org/project/holidays/" rel="nofollow">holiday detection package</a> and make sure we have it installed in our development environment as well.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->!echo <span class="hljs-string">"holidays"</span> >> requirements.txt | |
| !pip install -r requirements.txt<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1s8qwlb">Next, we have to adjust our <code>handler.py</code> and <code>EndpointHandler</code> to match our condition.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> typing <span class="hljs-keyword">import</span> <span class="hljs-type">Dict</span>, <span class="hljs-type">List</span>, <span class="hljs-type">Any</span> | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| <span class="hljs-keyword">import</span> holidays | |
| <span class="hljs-keyword">class</span> <span class="hljs-title class_">EndpointHandler</span>(): | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, path=<span class="hljs-string">""</span></span>): | |
| self.pipeline = pipeline(<span class="hljs-string">"text-classification"</span>,model=path) | |
| self.holidays = holidays.US() | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">__call__</span>(<span class="hljs-params">self, data: <span class="hljs-type">Dict</span>[<span class="hljs-built_in">str</span>, <span class="hljs-type">Any</span>]</span>) -> <span class="hljs-type">List</span>[<span class="hljs-type">Dict</span>[<span class="hljs-built_in">str</span>, <span class="hljs-type">Any</span>]]: | |
| <span class="hljs-string">""" | |
| data args: | |
| inputs (:obj: `str`) | |
| date (:obj: `str`) | |
| Return: | |
| A :obj:`list` | `dict`: will be serialized and returned | |
| """</span> | |
| <span class="hljs-comment"># get inputs</span> | |
| inputs = data.pop(<span class="hljs-string">"inputs"</span>,data) | |
| date = data.pop(<span class="hljs-string">"date"</span>, <span class="hljs-literal">None</span>) | |
| <span class="hljs-comment"># check if date exists and if it is a holiday</span> | |
| <span class="hljs-keyword">if</span> date <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span> <span class="hljs-keyword">and</span> date <span class="hljs-keyword">in</span> self.holidays: | |
| <span class="hljs-keyword">return</span> [{<span class="hljs-string">"label"</span>: <span class="hljs-string">"happy"</span>, <span class="hljs-string">"score"</span>: <span class="hljs-number">1</span>}] | |
| <span class="hljs-comment"># run normal prediction</span> | |
| prediction = self.pipeline(inputs) | |
| <span class="hljs-keyword">return</span> prediction<!-- HTML_TAG_END --></pre></div> <h3 class="relative group"><a id="4-test-endpointhandler" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#4-test-endpointhandler"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>4. Test EndpointHandler</span></h3> <p data-svelte-h="svelte-4es9m7">To test our EndpointHandler, we can simplify import, initialize and test it. Therefore we only need to prepare a sample payload.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> handler <span class="hljs-keyword">import</span> EndpointHandler | |
| <span class="hljs-comment"># init handler</span> | |
| my_handler = EndpointHandler(path=<span class="hljs-string">"."</span>) | |
| <span class="hljs-comment"># prepare sample payload</span> | |
| non_holiday_payload = {<span class="hljs-string">"inputs"</span>: <span class="hljs-string">"I am quite excited how this will turn out"</span>, <span class="hljs-string">"date"</span>: <span class="hljs-string">"2022-08-08"</span>} | |
| holiday_payload = {<span class="hljs-string">"inputs"</span>: <span class="hljs-string">"Today is a though day"</span>, <span class="hljs-string">"date"</span>: <span class="hljs-string">"2022-07-04"</span>} | |
| <span class="hljs-comment"># test the handler</span> | |
| non_holiday_pred=my_handler(non_holiday_payload) | |
| holiday_payload=my_handler(holiday_payload) | |
| <span class="hljs-comment"># show results</span> | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">"non_holiday_pred"</span>, non_holiday_pred) | |
| <span class="hljs-built_in">print</span>(<span class="hljs-string">"holiday_payload"</span>, holiday_payload) | |
| <span class="hljs-comment"># non_holiday_pred [{'label': 'joy', 'score': 0.9985942244529724}]</span> | |
| <span class="hljs-comment"># holiday_payload [{'label': 'happy', 'score': 1}]</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-vtyfqm">It works!!!! 🎉</p> <p data-svelte-h="svelte-15zdf0d"><em>Note: If you are using a notebook you might have to restart your kernel when you make changes to the handler.py since it is not automatically re-imported.</em></p> <h3 class="relative group"><a id="5-push-the-custom-handler-to-your-repository" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#5-push-the-custom-handler-to-your-repository"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>5. Push the Custom Handler to your repository</span></h3> <p data-svelte-h="svelte-1ii6eax">After you have successfully tested your handler locally, you can push it to your repository by simply using basic git commands.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --># <span class="hljs-built_in">add</span> <span class="hljs-keyword">all</span> our <span class="hljs-keyword">new</span> <span class="hljs-keyword">files</span> | |
| !git <span class="hljs-built_in">add</span> * | |
| # commit our <span class="hljs-keyword">files</span> | |
| !git commit -<span class="hljs-keyword">m</span> <span class="hljs-string">"add custom handler"</span> | |
| # push the <span class="hljs-keyword">files</span> <span class="hljs-keyword">to</span> the hub | |
| !git push<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-8xlhlw">Now, you should see your <code>handler.py</code> and <code>requirements.txt</code> in your repository in the <a href="https://huggingface.co/philschmid/distilbert-base-uncased-emotion/tree/main" rel="nofollow">“Files and version”</a> tab.</p> <h3 class="relative group"><a id="6-deploy-your-custom-handler-as-an-inference-endpoint" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#6-deploy-your-custom-handler-as-an-inference-endpoint"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>6. Deploy your Custom Handler as an Inference Endpoint</span></h3> <p data-svelte-h="svelte-n8fnak">The last step is to deploy your Custom Handler as an Inference Endpoint. You can deploy your Custom Handler like you would a regular Inference Endpoint. Add your repository, select your cloud and region, your instance and security setting, and deploy.</p> <p data-svelte-h="svelte-1uvy9xv">When creating your Endpoint, the Inference Endpoint Service will check for an available and valid <code>handler.py</code>, and will use it for serving requests no matter which “Task” you select.</p> <p data-svelte-h="svelte-hwhuk5"><em>Note: In your <a href="https://ui.endpoints.huggingface.co/" rel="nofollow">Inference Endpoints dashboard</a>, the Task for this Endpoint should now be set to Custom</em></p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/hf-endpoints-documentation/blob/main/docs/source/guides/custom_handler.mdx" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_87vzq7 = { | |
| assets: "/docs/inference-endpoints/pr_113/en", | |
| base: "/docs/inference-endpoints/pr_113/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/inference-endpoints/pr_113/en/_app/immutable/entry/start.d1c14968.js"), | |
| import("/docs/inference-endpoints/pr_113/en/_app/immutable/entry/app.18050d92.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 11], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 42.2 kB
- Xet hash:
- 0d6088fed6aa407191f9610f869d8bbc5157e79ec80d140570ea39d55599de80
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.