aws_summit_2022_inf1_bert_compile_and_deploy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AWS Summit San Francisco 2022\n",
    "## Using AWS Inferentia to optimize HuggingFace model inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to the AWS Summit San Francisco 2022 Inferentia Workshop! \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of contents\n",
    "1. [Introduction](#introduction)\n",
    "    1. [Setting up the environment](#setenv)\n",
    "3. [Get model from HuggingFace Model Hub](#getmodel)\n",
    "    1. [Get the Tokenizer](#gettoken)\n",
    "    2. [Download models and prepare them for inference](#trace)\n",
    "4. [Deploy default model to a CPU-based endpoint](#deploycpu)\n",
    "    1. [Perform a test CPU based inference](#testcpu)\n",
    "5. [Compile and deploy the model on an Inferentia instance](#compiledeploy)\n",
    "    1. [Review changes to the inference code](#reviewchanges)\n",
    "    2. [Create and compile Pytorch model for the inf1 instance](#pytorchmodel)\n",
    "    3. [Deploy compiled model into the inf1 instance](#deployinf1)\n",
    "    4. [Perform a test inf1 based inference](#testinf1)\n",
    "6. [Benchmark and comparison](#benchmark)\n",
    "    1. [Benchmark CPU based endpoint](#benchcpu)\n",
    "    2. [Benchmark Inferentia based endpoint](#benchinf1)\n",
    "7. [Comparison and conclusions](#conclusions)\n",
    "8. [Cleanup](#cleanup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction <a name=\"introduction\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During this workshop, we will create two endpoints with one HuggingFace model each. We will use them for the task of paraphrase detection which is an NLP classification problem. \n",
    "These two endpoints will have the following configurations: a) CPU-based endpoint, where we will be deploying the model with no changes; and b) Inf1 instance based endpoint, where we will prepare and compile the model using SageMaker Neo before deploying. \n",
    "Finally, we will perform a latency and throughput performance comparison of both endpoints. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[AWS Inferentia](https://aws.amazon.com/machine-learning/inferentia/) is Amazon's first ML chips designed to accelerate deep learning workloads and is part of a long-term strategy to deliver on this vision. AWS Inferentia is designed to provide high performance inference in the cloud, to drive down the total cost of inference, and to make it easy for developers to integrate machine learning into their business applications. AWS Inferentia chips deliver up 2.3x higher throughput and up to 70% lower cost per inference than comparable current generation GPU-based Amazon EC2 instances, as we will confirm in the example notebook.\n",
    "\n",
    "[AWS Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/) is a software development kit (SDK) for running machine learning inference using AWS Inferentia chips. It consists of a compiler, run-time, and profiling tools that enable developers to run high-performance and low latency inference using AWS Inferentia-based Amazon EC2 Inf1 instances. Using Neuron, you can bring your models that have been trained on any popular framework (PyTorch, TensorFlow, MXNet), and run them optimally on Inferentia. There is excellent support for Vision and NLP models especially, and on top of that we have released great features to help you make the most efficient use of the hardware, such as [dynamic batching](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/appnotes/perf/torch-neuron-dataparallel-app-note.html#dynamic-batching-description) or [Data Parallel](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-frameworks/pytorch-neuron/api-torch-neuron-dataparallel-api.html) inferencing.\n",
    "\n",
    "[SageMaker Neo](https://aws.amazon.com/sagemaker/neo/) saves you the effort of DIY model compilation, extending familiar SageMaker SDK API's to enable easy compilation for a [wide range](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_OutputConfig.html#API_OutputConfig_Contents) of platforms. This includes CPU and GPU-based instances, but also Inf1 instances; in this case, SageMaker Neo uses the Neuron SDK to compile your model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up the environment <a name=\"setenv\"></a>\n",
    "First, make sure you are using the Python 3 (Pytorch 1.8 Python 3.6 CPU Optimized) Kernel. And that you are working in the us-west-2 region unless instructed otherwise.\n",
    "\n",
    "Then, install ipywidgets library and restart the kernel to be able to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "import IPython\n",
    "import sys\n",
    "\n",
    "!{sys.executable} -m pip install ipywidgets\n",
    "IPython.Application.instance().kernel.do_shutdown(True)  # has to restart kernel so changes are used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "STOP! Restart the Kernel, comment the cell above and continue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will then install required Python packages. Also, we will create a default Amazon Sagemaker session, get the Amazon Sagemaker role and default Amazon S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install -U transformers\n",
    "!pip install -U sagemaker\n",
    "!pip install -U torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import transformers\n",
    "import sagemaker\n",
    "import torch\n",
    "import boto3\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "sess_bucket = sagemaker_session.default_bucket()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Get model from HuggingFace Model Hub <a name=\"getmodel\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this workshop, we will use [Prompsit/paraphrase-bert-en](https://huggingface.co/Prompsit/paraphrase-bert-en) transformer model from HuggingFace Model Hub. It has been fine-tuned from a pretrained model called \"bert-base-uncased\". The model works comparing a pair of sentences, it determines the semantic similarity between them. If the two sentences convey the same meaning it is labelled as paraphrase, otherwise it is labeled as non-paraphrase. \n",
    "So it allows to evaluate paraphrases for a given phrase, answering the following question: Is \"phrase B\" a paraphrase of \"phrase A\"? and the resulting probabilities correspond to classes:\n",
    "\n",
    "    0: Not a paraphrase\n",
    "    1: It's a paraphrase\n",
    "\n",
    "This model doesn't expect to find punctuation marks or long pieces of text.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the Tokenizer <a name=\"gettoken\"></a>\n",
    "As a first step, we need to get the tokenizer. A tokenizer breaks a stream of text into tokens, and it is in charge of preparing the inputs for a model. We need it to create a sample input to interact with the model, and will get it from HuggingFace through the `transformers` library. It is important to set the `return_dict` parameter to `False` when instantiating the model. In `transformers` v4.x, this parameter is `True` by default and it enables the return of dict-like python objects containing the model outputs, instead of the standard tuples. Neuron compilation does not support dictionary-based model ouputs, and compilation would fail if we didn't explictly set it to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = transformers.AutoTokenizer.from_pretrained(\"Prompsit/paraphrase-bert-en\")\n",
    "\n",
    "model = transformers.AutoModelForSequenceClassification.from_pretrained(\n",
    "    \"Prompsit/paraphrase-bert-en\", return_dict=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download models and prepare them for inference <a name=\"trace\"></a>\n",
    "We will download the model and create two files with different formats. The first one is the model itself with no changes. This one will be uploaded and used in the CPU based endpoint as it is. The second image is a traced Pytorch image of the model so we can compile it before deploying it to the inf1 instance.\n",
    "\n",
    "PyTorch models must be saved as a definition file (.pt or .pth) with input datatype of float32.\n",
    "To save the model, we will use torch.jit.trace followed by torch.save. This will save an object to a file ( a python pickle: pickle_module=pickle). \n",
    "\n",
    "Next, we will convert the saved model to a compressed tar file and upload it to an S3 bucket.\n",
    "As a final step, we will create a sample input to `jit.trace` of the model with PyTorch. We need this to have SageMaker Neo compile the model artifact.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Create directory for model artifacts\n",
    "Path(\"normal_model/\").mkdir(exist_ok=True)\n",
    "Path(\"traced_model/\").mkdir(exist_ok=True)\n",
    "\n",
    "# Prepare sample input for jit model tracing\n",
    "seq_0 = \"Welcome to AWS Summit San Francisco 2022! Thank you for attending the workshop on using Huggingface transformers on Inferentia instances.\"\n",
    "seq_1 = seq_0\n",
    "max_length = 512\n",
    "\n",
    "tokenized_sequence_pair = tokenizer.encode_plus(\n",
    "    seq_0, seq_1, max_length=max_length, padding=\"max_length\", truncation=True, return_tensors=\"pt\"\n",
    ")\n",
    "\n",
    "example = tokenized_sequence_pair[\"input_ids\"], tokenized_sequence_pair[\"attention_mask\"]\n",
    "\n",
    "traced_model = torch.jit.trace(model.eval(), example)\n",
    "\n",
    "model.save_pretrained('normal_model/')\n",
    "traced_model.save(\"traced_model/model.pth\") # The `.pth` extension is required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -czvf normal_model.tar.gz -C normal_model . && mv normal_model.tar.gz normal_model/\n",
    "!tar -czvf traced_model.tar.gz -C traced_model . && mv traced_model.tar.gz traced_model/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We upload the traced model `tar.gz` file to Amazon S3, where the compilation job will download it from"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "normal_model_url = sagemaker_session.upload_data(\n",
    "    path=\"normal_model/normal_model.tar.gz\",\n",
    "    key_prefix=\"neuron-experiments/bert-seq-classification/normal-model\",\n",
    ")\n",
    "\n",
    "traced_model_url = sagemaker_session.upload_data(\n",
    "    path=\"traced_model/traced_model.tar.gz\",\n",
    "    key_prefix=\"neuron-experiments/bert-seq-classification/traced-model\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Deploy default model to a CPU-based endpoint <a name=\"deploycpu\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a first step, we create model from the Hugging Face Model Class.\n",
    "We will be passing the `normal_model_url` as the `model_data` parameter to the `HuggingFaceModel` API. \n",
    "Notice that we are passing `inference.py` as the entry point script; also, the packages defined in the requirements file within the `source_dir` will automatically be installed in the endpoint instance. In this case we will use the `transformers` library that is compatible Inferentia instances (v. 4.15.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface import HuggingFaceModel\n",
    "from sagemaker.predictor import Predictor\n",
    "from datetime import datetime\n",
    "\n",
    "prefix = \"neuron-experiments/bert-seq-classification\"\n",
    "flavour = \"normal\"\n",
    "date_string = datetime.now().strftime(\"%Y%m-%d%H-%M%S\")\n",
    "\n",
    "normal_sm_model = HuggingFaceModel(\n",
    "    model_data=normal_model_url,\n",
    "    predictor_cls=Predictor,\n",
    "    transformers_version=\"4.12.3\",\n",
    "    pytorch_version='1.9.1',\n",
    "    role=role,\n",
    "    entry_point=\"inference.py\",\n",
    "    source_dir=\"code\",\n",
    "    py_version=\"py38\",\n",
    "    name=f\"{flavour}-distilbert-{date_string}\",\n",
    "    env={\"SAGEMAKER_CONTAINER_LOG_LEVEL\": \"10\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we create the endpoint and deploy the model for inference. This process will take about 4 minutes to complete. As you can see, one line of code will create a [real time endpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html) for real time inference that you can integrate to your applications. These endpoints are fully managed and support autoscaling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "\n",
    "hardware = \"c5\"\n",
    "\n",
    "normal_predictor = normal_sm_model.deploy(\n",
    "    instance_type=\"ml.c5.xlarge\",\n",
    "    initial_instance_count=1,\n",
    "    endpoint_name=f\"paraphrase-bert-en-{hardware}-{date_string}\",\n",
    "    serializer=JSONSerializer(),\n",
    "    deserializer=JSONDeserializer(),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform a test inference <a name=\"testcpu\"></a> on CPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will perform a quick test to see if the endpoint is responding as expected. We will send sample sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict with model endpoint\n",
    "client = boto3.client('sagemaker')\n",
    "\n",
    "#let's make sure it is up und running first\n",
    "status = \"\"\n",
    "while status != 'InService':\n",
    "    endpoint_response = client.describe_endpoint(EndpointName=f\"paraphrase-bert-en-{hardware}-{date_string}\")\n",
    "    status = endpoint_response['EndpointStatus']\n",
    "\n",
    "\n",
    "# Send a payload to the endpoint and recieve the inference\n",
    "payload = seq_0, seq_1\n",
    "normal_predictor.predict(payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Compile and deploy the model on an Inferentia instance <a name=\"compiledeploy\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will cover the compilation and deployment of the model into the inf1 instance. We will also review the changes in the inference code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review inference code <a name=\"reviewchanges\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you open `inference.py` you will see a few functions: \n",
    "a) `model_fn` which receives the model directory and is responsible for loading and returning the model.\n",
    "b) `input_fn` and `output_fn` functions that are in charge of pre-processing/checking content types of input and output to the endpoint.\n",
    "And c) `predict_fn`, receives the outputs of `model_fn` and `input_fn` and defines how the model will run inference (it recieves the loaded model and the deserialized/pre-processed input data).\n",
    "All of this code runs inside the endpoint once it is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize code/inference.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, notice that we will load the corresponding model depending on where the function is deployed. `model_fn` will return a tuple containing both the model and its corresponding tokenizer. Both the model and the input data will be sent `.to(device)`, which can be a CPU or GPU.\n",
    "\n",
    "Also, notice the `predict_fn`. In this function we recieve the string for inference, convert it to the format the model accepts, ask the model for the inference, recieve the inference and format it in clear text as a return string. In real life you might not need to do this interpretation since your application might be fine receiving the predicted class and use it directly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create and compile Pytorch model for the inf1 instance <a name=\"pytorchmodel\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now create a new `Huggingface` model that will use the `inference.py` file described above as its entry point script. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface import HuggingFaceModel\n",
    "from sagemaker.predictor import Predictor\n",
    "from datetime import datetime\n",
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "\n",
    "date_string = datetime.now().strftime(\"%Y%m-%d%H-%M%S\")\n",
    "hardware = \"inf1\"\n",
    "compilation_job_name = f\"paraphrase-bert-en-{hardware}-\" + date_string\n",
    "output_model_path = f\"s3://{sess_bucket}/{prefix}/neo-compilations19/{hardware}-model\"\n",
    "\n",
    "compiled_inf1_model = HuggingFaceModel(\n",
    "    model_data=traced_model_url,\n",
    "    predictor_cls=Predictor,\n",
    "    transformers_version=\"4.12.3\",\n",
    "    pytorch_version='1.9.1',\n",
    "    role=role,\n",
    "    entry_point=\"inference.py\",\n",
    "    source_dir=\"code\",\n",
    "    py_version=\"py37\",\n",
    "    name=f\"distilbert-{date_string}\",\n",
    "    env={\"SAGEMAKER_CONTAINER_LOG_LEVEL\": \"10\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to compile the model! Two additional notes:\n",
    "* HuggingFace models should be compiled to `dtype` `int64`\n",
    "* the format for `compiler_options` differs from the standard Python `dict` that you can use when compiling for \"normal\" instance types; for inferentia, you must provide a JSON string with CLI arguments, which correspond to the ones supported by the [Neuron Compiler](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-cc/command-line-reference.html) (read more about `compiler_options` [here](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_OutputConfig.html#API_OutputConfig_Contents))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model compilation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compile the model (this will take around 10 minutes to complete):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import json\n",
    "\n",
    "compiled_inf1_model = compiled_inf1_model.compile(\n",
    "    target_instance_family=f\"ml_{hardware}\",\n",
    "    input_shape={\"input_ids\": [1, 512], \"attention_mask\": [1, 512]},\n",
    "    job_name=compilation_job_name,\n",
    "    role=role,\n",
    "    framework=\"pytorch\",\n",
    "    framework_version=\"1.9.1\",\n",
    "    output_path=output_model_path,\n",
    "    compiler_options=json.dumps(\"--dtype int64\"),\n",
    "    compile_max_run=900,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compiler logs and artifacts\n",
    "Open a new browser tab and navigate to the Sagemaker Console. Under the Images menu on the left you will find the menu Inference and inside \"Compilation Jobs\". Here is where you will find the job that was executed in the previous cell. Look for the job name to get its details. If you scroll down you will find a section called \"Monitor\" you can access the compiler logs hosted in Cloudwatch. Look for the successful completion of the job in a line similar to the following:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "localhost compiler-container-Primary[4736]: Compiler status PASS\n",
    "and localhost compiler-container-Primary[4736]: INFO:Neuron:Neuron successfully compiled 1 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 100.0%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, in the Output section, you will find a link to the S3 compiled model artifact. Click on it so see where it was stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Compilation job name: {} \\nOutput model path in S3: {}\".format(compilation_job_name, output_model_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy compiled model into the inf1 instance <a name=\"deployinf1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After successful compilation, we deploy the new model to an inf1.xlarge instance based  [real time endpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html). As you can see, the one line of code procedure is similar to creating a CPU based instance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "compiled_inf1_predictor = compiled_inf1_model.deploy(\n",
    "    instance_type=\"ml.inf1.xlarge\",\n",
    "    initial_instance_count=1,\n",
    "    endpoint_name=f\"paraphrase-bert-en-{hardware}-{date_string}\",\n",
    "    serializer=JSONSerializer(),\n",
    "    deserializer=JSONDeserializer(),\n",
    "    wait=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform a test inference <a name=\"testinf1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a final test, we first make sure the endpoint is up una running in a `InService` state, and then perform a simple inference sending two sequences of text and wait for the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict with model endpoint\n",
    "client = boto3.client('sagemaker')\n",
    "\n",
    "#let's make sure it is up und running first\n",
    "status = \"\"\n",
    "while status != 'InService':\n",
    "    endpoint_response = client.describe_endpoint(EndpointName=f\"paraphrase-bert-en-{hardware}-{date_string}\")\n",
    "    status = endpoint_response['EndpointStatus']\n",
    "\n",
    "\n",
    "# Send a payload to the endpoint and recieve the inference\n",
    "payload = seq_0, seq_1\n",
    "compiled_inf1_predictor.predict(payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Benchmark and comparison <a name=\"benchmark\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have both endpoints online, we will perform a benchmark using Python's `threading` module. In each benchmark, we start 5 threads that will each make 100 requests to the model endpoint. We measure the inference latency for each request, and we also measure the total time to finish the task, so that we can get an estimate of the request throughput/second."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmark CPU based endpoint <a name=\"benchcpu\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# Run the benchmark \n",
    "\n",
    "import threading\n",
    "import time\n",
    "\n",
    "num_preds = 100\n",
    "num_threads = 5\n",
    "\n",
    "times = []\n",
    "\n",
    "\n",
    "def predict():\n",
    "    thread_id = threading.get_ident()\n",
    "    print(f\"Thread {thread_id} started\")\n",
    "\n",
    "    for i in range(num_preds):\n",
    "        tick = time.time()\n",
    "        response = normal_predictor.predict(payload)\n",
    "        tock = time.time()\n",
    "        times.append((thread_id, tock - tick))\n",
    "\n",
    "\n",
    "threads = []\n",
    "[threads.append(threading.Thread(target=predict, daemon=False)) for i in range(num_threads)]\n",
    "[t.start() for t in threads]\n",
    "\n",
    "# Wait for threads, get an estimate of total time\n",
    "start = time.time()\n",
    "[t.join() for t in threads]\n",
    "end = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display results \n",
    "from matplotlib.pyplot import hist, title, show, xlim\n",
    "import numpy as np\n",
    "\n",
    "TPS_CPU = (num_preds * num_threads) / end\n",
    "\n",
    "t_CPU = [duration for thread__id, duration in times]\n",
    "latency_percentiles = np.percentile(t_CPU, q=[50, 90, 95, 99])\n",
    "latency_CPU = latency_percentiles[2]*1000\n",
    "\n",
    "hist(t_CPU, bins=100)\n",
    "title(\"Request latency histogram on CPU\")\n",
    "show()\n",
    "\n",
    "print(\"==== Default HuggingFace model on CPU benchmark ====\\n\")\n",
    "print(f\"95 % of requests take less than {latency_CPU} ms\")\n",
    "print(f\"Rough request throughput/second is {TPS_CPU}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that request latency is in the 1-1.2 second range, and throughput is ~4.5 TPS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmark Inferentia based endpoint <a name=\"benchinf1\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# Run benchmark \n",
    "\n",
    "import threading\n",
    "import time\n",
    "\n",
    "\n",
    "num_preds = 300\n",
    "num_threads = 5\n",
    "\n",
    "times = []\n",
    "\n",
    "\n",
    "def predict():\n",
    "    thread_id = threading.get_ident()\n",
    "    print(f\"Thread {thread_id} started\")\n",
    "\n",
    "    for i in range(num_preds):\n",
    "        tick = time.time()\n",
    "        response = compiled_inf1_predictor.predict(payload)\n",
    "        tock = time.time()\n",
    "        times.append((thread_id, tock - tick))\n",
    "\n",
    "\n",
    "threads = []\n",
    "[threads.append(threading.Thread(target=predict, daemon=False)) for i in range(num_threads)]\n",
    "[t.start() for t in threads]\n",
    "\n",
    "# Make a rough estimate of total time, wait for threads\n",
    "start = time.time()\n",
    "[t.join() for t in threads]\n",
    "end = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display results \n",
    "from matplotlib.pyplot import hist, title, show, xlim\n",
    "import numpy as np\n",
    "\n",
    "TPS_inf1 = (num_preds * num_threads) / end\n",
    "\n",
    "t_inf1 = [duration for thread__id, duration in times]\n",
    "latency_percentiles = np.percentile(t_inf1, q=[50, 90, 95, 99])\n",
    "latency_inf1 = latency_percentiles[2]*1000\n",
    "\n",
    "hist(t_inf1, bins=100)\n",
    "title(\"Request latency histogram on Inferentia\")\n",
    "show()\n",
    "\n",
    "print(\"==== Default HuggingFace model on inf1 benchmark ====\\n\")\n",
    "print(f\"95 % of requests take less than {latency_inf1} ms\")\n",
    "print(f\"Rough request throughput/second is {TPS_inf1}\")\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that request latency is in the 0.02-0.05 millisecond range, and throughput is ~157 TPS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Conclusion <a name=\"conclusions\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Using inf1 instances latency dropped to a {:.2f} millisecond range from {:.2f} ms on a CPU endpoint.\".format(latency_inf1, latency_CPU)) \n",
    "print(\"Also, The average throughput increased to {:.2f} TPS from {:.2f} TPS on the CPU.\".format( TPS_inf1, TPS_CPU) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This increase in performance obtained from using inf1 instances, paired with the cost reduction and the use of known SageMaker SDK APIs, enables new benefits with little development effort and a gentle learning curve. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* To learn more about how to deploy Hugging Face modes through Sagemaker on to Inf1, please watch their latest [Webinar](https://www.youtube.com/watch?v=3fulTyMXhWQ), and read their latest [blog post](https://huggingface.co/blog/bert-inferentia-sagemaker). \n",
    "* For more information about Inferentia, please see the AWS EC2 Inf1 [website](https://aws.amazon.com/ec2/instance-types/inf1/) or check out other Tutorials available online [here] (https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-intro/tutorials.html).\n",
    "* You can learn more about Inferentia performance on the [Neuron Inference Performance](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/benchmark/index.html) pages\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Clean up <a name=\"cleanup\"></a>\n",
    "Delete the models and release the endpoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "normal_predictor.delete_model()\n",
    "normal_predictor.delete_endpoint()\n",
    "compiled_inf1_predictor.delete_model()\n",
    "compiled_inf1_predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.m5.large",
  "kernelspec": {
   "display_name": "Python 3 (PyTorch 1.8 Python 3.6 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/1.8.1-cpu-py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}