Duplicate from philschmid/roberta-base-squad2-optimized

Co-authored-by: Philipp Schmid <philschmid@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,32 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,292 @@
+---
+license: mit
+tags:
+- endpoints-template
+- optimum
+library_name: generic
+---
+
+# Optimized and Quantized [deepset/roberta-base-squad2](https://huggingface.co/deepset/roberta-base-squad2) with a custom handler.py
+
+
+This repository implements a `custom` handler for `question-answering` for 🤗 Inference Endpoints for accelerated inference using [🤗 Optiumum](https://huggingface.co/docs/optimum/index). The code for the customized handler is in the [handler.py](https://huggingface.co/philschmid/roberta-base-squad2-optimized/blob/main/handler.py).
+
+Below is also describe how we converted & optimized the model, based on the [Accelerate Transformers with Hugging Face Optimum](https://huggingface.co/blog/optimum-inference) blog post. You can also check out the [notebook](https://huggingface.co/philschmid/roberta-base-squad2-optimized/blob/main/optimize_model.ipynb).
+
+### expected Request payload
+
+```json
+{
+    "inputs": {
+        "question": "As what is Philipp working?", 
+        "context": "Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science. In the past I designed and implemented cloud-native machine learning architectures for fin-tech and insurance companies. I found my passion for cloud concepts and machine learning 5 years ago. Since then I never stopped learning. Currently, I am focusing myself in the area NLP and how to leverage models like BERT, Roberta, T5, ViT, and GPT2 to generate business value."
+    }
+}
+```
+
+below is an example on how to run a request using Python and `requests`.
+
+## Run Request 
+
+```python
+import json
+from typing import List
+import requests as r
+import base64
+
+ENDPOINT_URL = ""
+HF_TOKEN = ""
+
+
+def predict(question:str=None,context:str=None):
+    payload = {"inputs": {"question": question, "context": context}}
+    response = r.post(
+        ENDPOINT_URL, headers={"Authorization": f"Bearer {HF_TOKEN}"}, json=payload
+    )
+    return response.json()
+
+
+prediction = predict(
+    question="As what is Philipp working?",
+    context="Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science."
+)
+```
+
+expected output
+
+```python
+{
+    'score': 0.4749588668346405,
+    'start': 88,
+    'end': 102,
+    'answer': 'Technical Lead'
+}
+```
+
+
+
+# Convert & Optimize model with Optimum 
+
+Steps:
+1. [Convert model to ONNX](#1-convert-model-to-onnx)
+2. [Optimize & quantize model with Optimum](#2-optimize--quantize-model-with-optimum)
+3. [Create Custom Handler for Inference Endpoints](#3-create-custom-handler-for-inference-endpoints)
+4. [Test Custom Handler Locally](#4-test-custom-handler-locally)
+5. [Push to repository and create Inference Endpoint](#5-push-to-repository-and-create-inference-endpoint)
+
+Helpful links:
+* [Accelerate Transformers with Hugging Face Optimum](https://huggingface.co/blog/optimum-inference)
+* [Optimizing Transformers for GPUs with Optimum](https://www.philschmid.de/optimizing-transformers-with-optimum-gpu)
+* [Optimum Documentation](https://huggingface.co/docs/optimum/onnxruntime/modeling_ort)
+* [Create Custom Handler Endpoints](https://link-to-docs)
+
+## Setup & Installation
+
+
+```python
+%%writefile requirements.txt
+optimum[onnxruntime]==1.4.0
+mkl-include
+mkl
+```
+
+
+```python
+!pip install -r requirements.txt
+```
+
+## 0. Base line Performance
+
+
+```python
+from transformers import pipeline
+
+qa = pipeline("question-answering",model="deepset/roberta-base-squad2")
+```
+
+Okay, let's test the performance (latency) with sequence length of 128.
+
+
+```python
+context="Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science. In the past I designed and implemented cloud-native machine learning architectures for fin-tech and insurance companies. I found my passion for cloud concepts and machine learning 5 years ago. Since then I never stopped learning. Currently, I am focusing myself in the area NLP and how to leverage models like BERT, Roberta, T5, ViT, and GPT2 to generate business value." 
+question="As what is Philipp working?" 
+
+payload = {"inputs": {"question": question, "context": context}}
+```
+
+
+```python
+from time import perf_counter
+import numpy as np 
+
+def measure_latency(pipe,payload):
+    latencies = []
+    # warm up
+    for _ in range(10):
+        _ = pipe(question=payload["inputs"]["question"], context=payload["inputs"]["context"])
+    # Timed run
+    for _ in range(50):
+        start_time = perf_counter()
+        _ =  pipe(question=payload["inputs"]["question"], context=payload["inputs"]["context"])
+        latency = perf_counter() - start_time
+        latencies.append(latency)
+    # Compute run statistics
+    time_avg_ms = 1000 * np.mean(latencies)
+    time_std_ms = 1000 * np.std(latencies)
+    return f"Average latency (ms) - {time_avg_ms:.2f} +\- {time_std_ms:.2f}"
+
+print(f"Vanilla model {measure_latency(qa,payload)}")
+#     Vanilla model Average latency (ms) - 64.15 +\- 2.44
+```
+
+
+
+## 1. Convert model to ONNX
+
+
+```python
+from optimum.onnxruntime import ORTModelForQuestionAnswering
+from transformers import AutoTokenizer
+from pathlib import Path
+
+
+model_id="deepset/roberta-base-squad2"
+onnx_path = Path(".")
+
+# load vanilla transformers and convert to onnx
+model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True)
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+# save onnx checkpoint and tokenizer
+model.save_pretrained(onnx_path)
+tokenizer.save_pretrained(onnx_path)
+```
+
+
+## 2. Optimize & quantize model with Optimum
+
+
+```python
+from optimum.onnxruntime import ORTOptimizer, ORTQuantizer
+from optimum.onnxruntime.configuration import OptimizationConfig, AutoQuantizationConfig
+
+# Create the optimizer
+optimizer = ORTOptimizer.from_pretrained(model)
+
+# Define the optimization strategy by creating the appropriate configuration
+optimization_config = OptimizationConfig(optimization_level=99) # enable all optimizations
+
+# Optimize the model
+optimizer.optimize(save_dir=onnx_path, optimization_config=optimization_config)
+```
+
+
+```python
+# create ORTQuantizer and define quantization configuration
+dynamic_quantizer = ORTQuantizer.from_pretrained(onnx_path, file_name="model_optimized.onnx")
+dqconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)
+
+# apply the quantization configuration to the model
+model_quantized_path = dynamic_quantizer.quantize(
+    save_dir=onnx_path,
+    quantization_config=dqconfig,
+)
+
+```
+
+## 3. Create Custom Handler for Inference Endpoints
+
+
+
+```python
+%%writefile handler.py
+from typing import  Dict, List, Any
+from optimum.onnxruntime import ORTModelForQuestionAnswering
+from transformers import AutoTokenizer, pipeline
+
+
+class EndpointHandler():
+    def __init__(self, path=""):
+        # load the optimized model
+        self.model = ORTModelForQuestionAnswering.from_pretrained(path, file_name="model_optimized_quantized.onnx")
+        self.tokenizer = AutoTokenizer.from_pretrained(path)
+        # create pipeline
+        self.pipeline = pipeline("question-answering", model=self.model, tokenizer=self.tokenizer)
+
+    def __call__(self, data: Any) -> List[List[Dict[str, float]]]:
+        """
+        Args:
+            data (:obj:):
+                includes the input data and the parameters for the inference.
+        Return:
+            A :obj:`list`:. The list contains the answer and scores of the inference inputs
+        """
+        inputs = data.get("inputs", data)
+        # run the model
+        prediction = self.pipeline(**inputs)
+        # return prediction
+        return prediction
+```
+
+## 4. Test Custom Handler Locally
+
+
+
+```python
+from handler import EndpointHandler
+
+# init handler
+my_handler = EndpointHandler(path=".")
+
+# prepare sample payload
+context="Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science. In the past I designed and implemented cloud-native machine learning architectures for fin-tech and insurance companies. I found my passion for cloud concepts and machine learning 5 years ago. Since then I never stopped learning. Currently, I am focusing myself in the area NLP and how to leverage models like BERT, Roberta, T5, ViT, and GPT2 to generate business value." 
+question="As what is Philipp working?" 
+
+payload = {"inputs": {"question": question, "context": context}}
+
+# test the handler
+my_handler(payload)
+```
+
+
+```python
+from time import perf_counter
+import numpy as np 
+
+def measure_latency(handler,payload):
+    latencies = []
+    # warm up
+    for _ in range(10):
+        _ = handler(payload)
+    # Timed run
+    for _ in range(50):
+        start_time = perf_counter()
+        _ =  handler(payload)
+        latency = perf_counter() - start_time
+        latencies.append(latency)
+    # Compute run statistics
+    time_avg_ms = 1000 * np.mean(latencies)
+    time_std_ms = 1000 * np.std(latencies)
+    return f"Average latency (ms) - {time_avg_ms:.2f} +\- {time_std_ms:.2f}"
+
+print(f"Optimized & Quantized model {measure_latency(my_handler,payload)}")
+#     
+
+```
+
+`Optimized & Quantized model Average latency (ms) - 29.90 +\- 0.53`
+`Vanilla model Average latency (ms) - 64.15 +\- 2.44`
+
+## 5. Push to repository and create Inference Endpoint
+
+
+
+```python
+# add all our new files
+!git add * 
+# commit our files
+!git commit -m "add custom handler"
+# push the files to the hub
+!git push
+```
+

config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "deepset/roberta-base-squad2",
+  "architectures": [
+    "RobertaForQuestionAnswering"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "language": "english",
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "name": "Roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.21.3",
+  "type_vocab_size": 1,
+  "use_cache": false,
+  "vocab_size": 50265
+}

handler.py

@@ -0,0 +1,26 @@
+from typing import  Dict, List, Any
+from optimum.onnxruntime import ORTModelForQuestionAnswering
+from transformers import AutoTokenizer, pipeline
+
+
+class EndpointHandler():
+    def __init__(self, path=""):
+        # load the optimized model
+        self.model = ORTModelForQuestionAnswering.from_pretrained(path, file_name="model_optimized_quantized.onnx")
+        self.tokenizer = AutoTokenizer.from_pretrained(path)
+        # create pipeline
+        self.pipeline = pipeline("question-answering", model=self.model, tokenizer=self.tokenizer)
+
+    def __call__(self, data: Any) -> List[List[Dict[str, float]]]:
+        """
+        Args:
+            data (:obj:):
+                includes the input data and the parameters for the inference.
+        Return:
+            A :obj:`list`:. The list contains the answer and scores of the inference inputs
+        """
+        inputs = data.get("inputs", data)
+        # run the model
+        prediction = self.pipeline(**inputs)
+        # return prediction
+        return prediction

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:570afefbc8642150310e46c10a252dd091c8f44449e8a3a65a425f77991dc2ab
+size 496337664

model_optimized.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11c0577c4bb3afdb2a88e21807d5722511b6aa678d6d8275a7ba73c5cd8f88b1
+size 496254364

model_optimized_quantized.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a27adda924cc0cd34fde41f606da51673ebefb7132a5518e41f1196ebc362f1
+size 305175132

optimize_model.ipynb

@@ -0,0 +1,480 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Convert & Optimize model with Optimum \n",
+    "\n",
+    "\n",
+    "Steps:\n",
+    "1. Convert model to ONNX\n",
+    "2. Optimize & quantize model with Optimum\n",
+    "3. Create Custom Handler for Inference Endpoints\n",
+    "4. Test Custom Handler Locally\n",
+    "5. Push to repository and create Inference Endpoint\n",
+    "\n",
+    "Helpful links:\n",
+    "* [Accelerate Transformers with Hugging Face Optimum](https://huggingface.co/blog/optimum-inference)\n",
+    "* [Optimizing Transformers for GPUs with Optimum](https://www.philschmid.de/optimizing-transformers-with-optimum-gpu)\n",
+    "* [Optimum Documentation](https://huggingface.co/docs/optimum/onnxruntime/modeling_ort)\n",
+    "* [Create Custom Handler Endpoints](https://link-to-docs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup & Installation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Writing requirements.txt\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile requirements.txt\n",
+    "optimum[onnxruntime]==1.4.0\n",
+    "mkl-include\n",
+    "mkl"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install -r requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Base line Performance\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import pipeline\n",
+    "\n",
+    "qa = pipeline(\"question-answering\",model=\"deepset/roberta-base-squad2\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Okay, let's test the performance (latency) with sequence length of 128."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'{\"inputs\": {\"question\": \"As what is Philipp working?\", \"context\": \"Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science. In the past I designed and implemented cloud-native machine learning architectures for fin-tech and insurance companies. I found my passion for cloud concepts and machine learning 5 years ago. Since then I never stopped learning. Currently, I am focusing myself in the area NLP and how to leverage models like BERT, Roberta, T5, ViT, and GPT2 to generate business value.\"}}'"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "context=\"Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science. In the past I designed and implemented cloud-native machine learning architectures for fin-tech and insurance companies. I found my passion for cloud concepts and machine learning 5 years ago. Since then I never stopped learning. Currently, I am focusing myself in the area NLP and how to leverage models like BERT, Roberta, T5, ViT, and GPT2 to generate business value.\" \n",
+    "question=\"As what is Philipp working?\" \n",
+    "\n",
+    "payload = {\"inputs\": {\"question\": question, \"context\": context}}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Vanilla model Average latency (ms) - 64.15 +\\- 2.44\n"
+     ]
+    }
+   ],
+   "source": [
+    "from time import perf_counter\n",
+    "import numpy as np \n",
+    "\n",
+    "def measure_latency(pipe,payload):\n",
+    "    latencies = []\n",
+    "    # warm up\n",
+    "    for _ in range(10):\n",
+    "        _ = pipe(question=payload[\"inputs\"][\"question\"], context=payload[\"inputs\"][\"context\"])\n",
+    "    # Timed run\n",
+    "    for _ in range(50):\n",
+    "        start_time = perf_counter()\n",
+    "        _ =  pipe(question=payload[\"inputs\"][\"question\"], context=payload[\"inputs\"][\"context\"])\n",
+    "        latency = perf_counter() - start_time\n",
+    "        latencies.append(latency)\n",
+    "    # Compute run statistics\n",
+    "    time_avg_ms = 1000 * np.mean(latencies)\n",
+    "    time_std_ms = 1000 * np.std(latencies)\n",
+    "    return f\"Average latency (ms) - {time_avg_ms:.2f} +\\- {time_std_ms:.2f}\"\n",
+    "\n",
+    "print(f\"Vanilla model {measure_latency(qa,payload)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Convert model to ONNX"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "df00c03d67b546bf8a3d1a327b9380f5",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading:   0%|          | 0.00/571 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "('./tokenizer_config.json',\n",
+       " './special_tokens_map.json',\n",
+       " './vocab.json',\n",
+       " './merges.txt',\n",
+       " './added_tokens.json',\n",
+       " './tokenizer.json')"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from optimum.onnxruntime import ORTModelForQuestionAnswering\n",
+    "from transformers import AutoTokenizer\n",
+    "from pathlib import Path\n",
+    "\n",
+    "\n",
+    "model_id=\"deepset/roberta-base-squad2\"\n",
+    "onnx_path = Path(\".\")\n",
+    "\n",
+    "# load vanilla transformers and convert to onnx\n",
+    "model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True)\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
+    "\n",
+    "# save onnx checkpoint and tokenizer\n",
+    "model.save_pretrained(onnx_path)\n",
+    "tokenizer.save_pretrained(onnx_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Optimize & quantize model with Optimum"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "2022-09-12 18:47:03.240390005 [W:onnxruntime:, inference_session.cc:1488 Initialize] Serializing optimized model with Graph Optimization level greater than ORT_ENABLE_EXTENDED and the NchwcTransformer enabled. The generated model may contain hardware specific optimizations, and should only be used in the same environment the model was optimized in.\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "PosixPath('.')"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from optimum.onnxruntime import ORTOptimizer, ORTQuantizer\n",
+    "from optimum.onnxruntime.configuration import OptimizationConfig, AutoQuantizationConfig\n",
+    "\n",
+    "# Create the optimizer\n",
+    "optimizer = ORTOptimizer.from_pretrained(model)\n",
+    "\n",
+    "# Define the optimization strategy by creating the appropriate configuration\n",
+    "optimization_config = OptimizationConfig(optimization_level=99) # enable all optimizations\n",
+    "\n",
+    "# Optimize the model\n",
+    "optimizer.optimize(save_dir=onnx_path, optimization_config=optimization_config)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create ORTQuantizer and define quantization configuration\n",
+    "dynamic_quantizer = ORTQuantizer.from_pretrained(onnx_path, file_name=\"model_optimized.onnx\")\n",
+    "dqconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)\n",
+    "\n",
+    "# apply the quantization configuration to the model\n",
+    "model_quantized_path = dynamic_quantizer.quantize(\n",
+    "    save_dir=onnx_path,\n",
+    "    quantization_config=dqconfig,\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Create Custom Handler for Inference Endpoints\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting handler.py\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile handler.py\n",
+    "from typing import  Dict, List, Any\n",
+    "from optimum.onnxruntime import ORTModelForQuestionAnswering\n",
+    "from transformers import AutoTokenizer, pipeline\n",
+    "\n",
+    "\n",
+    "class EndpointHandler():\n",
+    "    def __init__(self, path=\"\"):\n",
+    "        # load the optimized model\n",
+    "        self.model = ORTModelForQuestionAnswering.from_pretrained(path, file_name=\"model_optimized_quantized.onnx\")\n",
+    "        self.tokenizer = AutoTokenizer.from_pretrained(path)\n",
+    "        # create pipeline\n",
+    "        self.pipeline = pipeline(\"question-answering\", model=self.model, tokenizer=self.tokenizer)\n",
+    "\n",
+    "    def __call__(self, data: Any) -> List[List[Dict[str, float]]]:\n",
+    "        \"\"\"\n",
+    "        Args:\n",
+    "            data (:obj:):\n",
+    "                includes the input data and the parameters for the inference.\n",
+    "        Return:\n",
+    "            A :obj:`list`:. The list contains the answer and scores of the inference inputs\n",
+    "        \"\"\"\n",
+    "        inputs = data.get(\"inputs\", data)\n",
+    "        # run the model\n",
+    "        prediction = self.pipeline(**inputs)\n",
+    "        # return prediction\n",
+    "        return prediction"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Test Custom Handler Locally\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'score': 0.4749588668346405,\n",
+       " 'start': 88,\n",
+       " 'end': 102,\n",
+       " 'answer': 'Technical Lead'}"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from handler import EndpointHandler\n",
+    "\n",
+    "# init handler\n",
+    "my_handler = EndpointHandler(path=\".\")\n",
+    "\n",
+    "# prepare sample payload\n",
+    "context=\"Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science. In the past I designed and implemented cloud-native machine learning architectures for fin-tech and insurance companies. I found my passion for cloud concepts and machine learning 5 years ago. Since then I never stopped learning. Currently, I am focusing myself in the area NLP and how to leverage models like BERT, Roberta, T5, ViT, and GPT2 to generate business value.\" \n",
+    "question=\"As what is Philipp working?\" \n",
+    "\n",
+    "payload = {\"inputs\": {\"question\": question, \"context\": context}}\n",
+    "\n",
+    "# test the handler\n",
+    "my_handler(payload)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Optimized & Quantized model Average latency (ms) - 29.90 +\\- 0.53\n"
+     ]
+    }
+   ],
+   "source": [
+    "from time import perf_counter\n",
+    "import numpy as np \n",
+    "\n",
+    "def measure_latency(handler,payload):\n",
+    "    latencies = []\n",
+    "    # warm up\n",
+    "    for _ in range(10):\n",
+    "        _ = handler(payload)\n",
+    "    # Timed run\n",
+    "    for _ in range(50):\n",
+    "        start_time = perf_counter()\n",
+    "        _ =  handler(payload)\n",
+    "        latency = perf_counter() - start_time\n",
+    "        latencies.append(latency)\n",
+    "    # Compute run statistics\n",
+    "    time_avg_ms = 1000 * np.mean(latencies)\n",
+    "    time_std_ms = 1000 * np.std(latencies)\n",
+    "    return f\"Average latency (ms) - {time_avg_ms:.2f} +\\- {time_std_ms:.2f}\"\n",
+    "\n",
+    "print(f\"Optimized & Quantized model {measure_latency(my_handler,payload)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`Vanilla model Average latency (ms) - 64.15 +\\- 2.44`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Push to repository and create Inference Endpoint\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[main a854397] add custom handler\n",
+      " 14 files changed, 151227 insertions(+)\n",
+      " create mode 100644 README.md\n",
+      " create mode 100644 config.json\n",
+      " create mode 100644 handler.py\n",
+      " create mode 100644 merges.txt\n",
+      " create mode 100644 model.onnx\n",
+      " create mode 100644 model_optimized.onnx\n",
+      " create mode 100644 model_optimized_quantized.onnx\n",
+      " create mode 100644 optimize_model.ipynb\n",
+      " create mode 100644 ort_config.json\n",
+      " create mode 100644 requirements.txt\n",
+      " create mode 100644 special_tokens_map.json\n",
+      " create mode 100644 tokenizer.json\n",
+      " create mode 100644 tokenizer_config.json\n",
+      " create mode 100644 vocab.json\n",
+      "Username for 'https://huggingface.co': ^C\n"
+     ]
+    }
+   ],
+   "source": [
+    "# add all our new files\n",
+    "!git add * \n",
+    "# commit our files\n",
+    "!git commit -m \"add custom handler\"\n",
+    "# push the files to the hub\n",
+    "!git push"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.9.12 ('az': conda)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.12"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "bddb99ecda5b40a820d97bf37f3ff3a89fb9dbcf726ae84d28624ac628a665b4"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

ort_config.json

@@ -0,0 +1,29 @@
+{
+  "opset": null,
+  "optimization": {},
+  "optimum_version": "1.4.0",
+  "quantization": {
+    "activations_dtype": "QUInt8",
+    "activations_symmetric": false,
+    "format": "QOperator",
+    "is_static": false,
+    "mode": "IntegerOps",
+    "nodes_to_exclude": [],
+    "nodes_to_quantize": [],
+    "operators_to_quantize": [
+      "MatMul",
+      "Add"
+    ],
+    "per_channel": false,
+    "qdq_add_pair_to_weight": false,
+    "qdq_dedicated_pair": false,
+    "qdq_op_type_per_channel_support_to_axis": {
+      "MatMul": 1
+    },
+    "reduce_range": false,
+    "weights_dtype": "QInt8",
+    "weights_symmetric": true
+  },
+  "transformers_version": "4.21.3",
+  "use_external_data_format": false
+}

requirements.txt

@@ -0,0 +1,3 @@
+optimum[onnxruntime]==1.4.0
+mkl-include
+mkl

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,67 @@
+{
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "do_lower_case": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "errors": "replace",
+  "full_tokenizer_file": null,
+  "mask_token": {
+    "__type": "AddedToken",
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "model_max_length": 512,
+  "name_or_path": "deepset/roberta-base-squad2",
+  "pad_token": {
+    "__type": "AddedToken",
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "special_tokens_map_file": "/home/ubuntu/.cache/huggingface/transformers/c9d2c178fac8d40234baa1833a3b1903d393729bf93ea34da247c07db24900d0.cb2244924ab24d706b02fd7fcedaea4531566537687a539ebb94db511fd122a0",
+  "tokenizer_class": "RobertaTokenizer",
+  "trim_offsets": true,
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}