introduction_to_amazon_algorithms/jumpstart_instance_segmentation/Amazon_JumpStart_Instance_Segmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6501d8ff",
   "metadata": {},
   "source": [
    "# Introduction to JumpStart - Instance Segmentation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7dcfa2da",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f6bdb0",
   "metadata": {},
   "source": [
    "---\n",
    "Welcome to Amazon [SageMaker JumpStart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html)! You can use JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through [SageMaker JumpStart API](https://sagemaker.readthedocs.io/en/stable/doc_utils/jumpstart.html). In this demo notebook, we demonstrate how to use the JumpStart API for Instance Segmentation. Instance segmentation is the task of detecting and delineating each distinct object of interest appearing in an image. It is a fine-grained, pixel-level approach to developing computer vision applications. It tags every pixel in an image with a class label from a predefined set of classes. It differs from Semantic Segmentation in the following: Instance Segmentation treats multiple objects of the same class as distinct individual instances whereas Semantic Segmentation treats multiple objects of the same class as a single entity.\n",
    "\n",
    "\n",
    "In this notebook, we demonstrate how to use pre-trained Instance Segmentation models for inference.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63f4c516",
   "metadata": {},
   "source": [
    "1. [Set Up](#1.-Set-Up)\n",
    "2. [Select a pre-trained model](#2.-Select-a-pre-trained-model)\n",
    "3. [Run inference on the pre-trained model](#3.-Run-inference-on-the-pre-trained-model)\n",
    "    * [Retrieve JumpStart Artifacts & Deploy an Endpoint](#3.1.-Retrieve-JumpStart-Artifacts-&-Deploy-an-Endpoint)\n",
    "    * [Download an example image for inference](#3.2.-Download-an-example-image-for-inference)\n",
    "    * [Query endpoint and parse response](#3.3.-Query-endpoint-and-parse-response)\n",
    "    * [Display model predictions](#3.4.-Display-model-predictions)\n",
    "    * [Clean up the endpoint](#3.5.-Clean-up-the-endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aecf218f",
   "metadata": {},
   "source": [
    "Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bc55dad",
   "metadata": {},
   "source": [
    "## 1. Set Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bd333a5",
   "metadata": {},
   "source": [
    "---\n",
    "Before executing the notebook, there are some initial steps required for set up. This notebook requires latest version of sagemaker and ipywidgets\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26b34b1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sagemaker ipywidgets --upgrade --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "395cfd4b",
   "metadata": {},
   "source": [
    "### Permissions and environment variables\n",
    "\n",
    "---\n",
    "To train and host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook as the AWS account role with SageMaker access. It has necessary permissions, including access to your data in S3. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c75cfa7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker, boto3, json\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "aws_role = get_execution_role()\n",
    "aws_region = boto3.Session().region_name\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69c0405",
   "metadata": {},
   "source": [
    "### 2. Select a pre-trained model\n",
    "\n",
    "***\n",
    "Here, we download jumpstart model_manifest file from the jumpstart s3 bucket, filter-out all the Instance Segmentation models and select a model for inference.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9455b70c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import Dropdown\n",
    "\n",
    "# download JumpStart model_manifest file.\n",
    "boto3.client(\"s3\").download_file(\n",
    "    f\"jumpstart-cache-prod-{aws_region}\", \"models_manifest.json\", \"models_manifest.json\"\n",
    ")\n",
    "with open(\"models_manifest.json\", \"rb\") as json_file:\n",
    "    model_list = json.load(json_file)\n",
    "\n",
    "# filter-out all the Instance Segmentation models from the manifest list.\n",
    "is_models = []\n",
    "for model in model_list:\n",
    "    model_id = model[\"model_id\"]\n",
    "    if \"-is-\" in model_id and model_id not in is_models:\n",
    "        is_models.append(model_id)\n",
    "\n",
    "print(f\"\\033[38;5;2mChose a model for inference: \\033[0;0m\\n\")\n",
    "\n",
    "# display the model-ids in a dropdown to select a model for inference.\n",
    "model_dropdown = Dropdown(\n",
    "    options=is_models,\n",
    "    value=\"mxnet-is-mask-rcnn-fpn-resnet101-v1d-coco\",\n",
    "    description=\"\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")\n",
    "display(model_dropdown)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "992d0eaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_version=\"*\" fetches the latest version of the model\n",
    "model_id, model_version = model_dropdown.value, \"*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dd62c0a",
   "metadata": {},
   "source": [
    "## 3. Run inference on the pre-trained model\n",
    "\n",
    "***\n",
    "\n",
    "Using JumpStart, we can perform inference on the pre-trained model, even without fine-tuning it first on a new dataset. \n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cd41e41",
   "metadata": {},
   "source": [
    "### 3.1. Retrieve JumpStart Artifacts & Deploy an Endpoint\n",
    "\n",
    "***\n",
    "We retrieve the `deploy_image_uri`, `deploy_source_uri`, and `base_model_uri` for the pre-trained model. To host the pre-trained base-model, we create an instance of [`sagemaker.model.Model`](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html) and deploy it.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c855c2e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import image_uris, model_uris, script_uris, hyperparameters\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "\n",
    "endpoint_name = name_from_base(f\"jumpstart-example-infer-{model_id}\")\n",
    "\n",
    "inference_instance_type = \"ml.p2.xlarge\"\n",
    "\n",
    "# Retrieve the inference docker container uri\n",
    "deploy_image_uri = image_uris.retrieve(\n",
    "    region=None,\n",
    "    framework=None,  # automatically inferred from model_id\n",
    "    image_scope=\"inference\",\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    instance_type=inference_instance_type,\n",
    ")\n",
    "\n",
    "# Retrieve the inference script uri. This includes scripts for model loading, inference handling etc.\n",
    "deploy_source_uri = script_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, script_scope=\"inference\"\n",
    ")\n",
    "\n",
    "\n",
    "# Retrieve the base model uri\n",
    "base_model_uri = model_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, model_scope=\"inference\"\n",
    ")\n",
    "\n",
    "\n",
    "# Create the SageMaker model instance\n",
    "model = Model(\n",
    "    image_uri=deploy_image_uri,\n",
    "    source_dir=deploy_source_uri,\n",
    "    model_data=base_model_uri,\n",
    "    entry_point=\"inference.py\",  # entry point file in source_dir and present in deploy_source_uri\n",
    "    role=aws_role,\n",
    "    predictor_cls=Predictor,\n",
    "    name=endpoint_name,\n",
    ")\n",
    "\n",
    "# deploy the Model. Note that we need to pass Predictor class when we deploy model through Model class,\n",
    "# for being able to run inference through the sagemaker API.\n",
    "base_model_predictor = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=inference_instance_type,\n",
    "    predictor_cls=Predictor,\n",
    "    endpoint_name=endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "026fd90d",
   "metadata": {},
   "source": [
    "### 3.2. Download an example image for inference\n",
    "---\n",
    "We download an example image from the JumpStart assets S3 bucket.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e301c74f",
   "metadata": {},
   "outputs": [],
   "source": [
    "jumpstart_assets_s3_bucket = f\"jumpstart-cache-prod-{aws_region}\"\n",
    "pedestrian_img_key_prefix = \"inference-notebook-assets\"\n",
    "pedestrian_img = \"img_pedestrian.png\"\n",
    "\n",
    "boto3.client(\"s3\").download_file(\n",
    "    jumpstart_assets_s3_bucket, f\"{pedestrian_img_key_prefix}/{pedestrian_img}\", pedestrian_img\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97d7a8c2",
   "metadata": {},
   "source": [
    "### 3.3. Query endpoint and parse response\n",
    "\n",
    "---\n",
    "Input to the endpoint is a single image in binary format. Response of the endpoint is a set of bounding boxes, masks, class names and scores for the predictions. Endpoint also returns the image overlaid with the mask. JumpStart allows the flexibility in the number of predictions returned. Below, we show to predict two objects per image by appending `;n_predictions=2` to Accept. To predict xx boxes, one can change it to `;n_predictions=xx` or to get all the predicted boxes, one can remove `;n_predictions=2`. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d9d30a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "def query(model_predictor, image_file_name):\n",
    "    \"\"\"Query the model predictor.\"\"\"\n",
    "\n",
    "    with open(image_file_name, \"rb\") as file:\n",
    "        input_img_rb = file.read()\n",
    "\n",
    "    query_response = model_predictor.predict(\n",
    "        input_img_rb,\n",
    "        {\n",
    "            \"ContentType\": \"application/x-image\",\n",
    "            \"Accept\": \"application/json;verbose;n_predictions=2\",\n",
    "        },\n",
    "    )\n",
    "    return query_response\n",
    "\n",
    "\n",
    "def parse_response(query_response):\n",
    "    \"\"\"Parse response and return a set of bounding boxes, masks, class names and scores for predictions along with the original image overlaid with the mask.\"\"\"\n",
    "\n",
    "    response_dict = json.loads(query_response)\n",
    "    return (\n",
    "        response_dict[\"ids\"],\n",
    "        response_dict[\"scores\"],\n",
    "        response_dict[\"bboxes\"],\n",
    "        response_dict[\"masks\"],\n",
    "        response_dict[\"labels\"],\n",
    "        response_dict[\"image_with_masks\"],\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b27d855",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_response = query(base_model_predictor, pedestrian_img)\n",
    "ids, scores, bboxes, masks, labels, image_with_masks = parse_response(query_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cdc6da2",
   "metadata": {},
   "source": [
    "### 3.4. Display model predictions\n",
    "---\n",
    "Next, we to plot the boxes on top of image with masks. For this, we adopt a similar function from [GluonCV](https://cv.gluon.ai/_modules/gluoncv/utils/viz/bbox.html#plot_bbox).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "debd31ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_bbox(\n",
    "    img,\n",
    "    bboxes,\n",
    "    scores=None,\n",
    "    labels=None,\n",
    "    thresh=0.5,\n",
    "    class_names=None,\n",
    "    colors=None,\n",
    "    ax=None,\n",
    "    linewidth=3.5,\n",
    "    fontsize=12,\n",
    "):\n",
    "    \"\"\"Plot box over the predicted objects.\"\"\"\n",
    "\n",
    "    from matplotlib import pyplot as plt\n",
    "    import random\n",
    "\n",
    "    img = img.copy()\n",
    "    ax.imshow(img.astype(np.uint8))\n",
    "\n",
    "    colors = dict()\n",
    "\n",
    "    for i, bbox in enumerate(bboxes):\n",
    "        if scores.flat[i] < thresh or labels.flat[i] < 0:\n",
    "            continue\n",
    "        cls_id = int(labels.flat[i]) if labels is not None else -1\n",
    "        if cls_id not in colors:\n",
    "            if class_names is not None:\n",
    "                colors[cls_id] = plt.get_cmap(\"hsv\")(cls_id / len(class_names))\n",
    "            else:\n",
    "                colors[cls_id] = (random.random(), random.random(), random.random())\n",
    "        xmin, ymin, xmax, ymax = [int(x) for x in bbox]\n",
    "        rect = plt.Rectangle(\n",
    "            (xmin, ymin),\n",
    "            xmax - xmin,\n",
    "            ymax - ymin,\n",
    "            fill=False,\n",
    "            edgecolor=colors[cls_id],\n",
    "            linewidth=linewidth,\n",
    "        )\n",
    "        ax.add_patch(rect)\n",
    "        if class_names is not None and cls_id < len(class_names):\n",
    "            class_name = class_names[cls_id]\n",
    "        else:\n",
    "            class_name = str(cls_id) if cls_id >= 0 else \"\"\n",
    "        score = \"{:.3f}\".format(scores.flat[i]) if scores is not None else \"\"\n",
    "        if class_name or score:\n",
    "            ax.text(\n",
    "                xmin,\n",
    "                ymin - 2,\n",
    "                \"{:s} {:s}\".format(class_name, score),\n",
    "                bbox=dict(facecolor=colors[cls_id], alpha=0.5),\n",
    "                fontsize=fontsize,\n",
    "                color=\"white\",\n",
    "            )\n",
    "    return ax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "188bde73",
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import numpy as np\n",
    "\n",
    "ids, scores, bboxes, masks, image_with_masks = (\n",
    "    np.array(ids),\n",
    "    np.array(scores),\n",
    "    np.array(bboxes),\n",
    "    np.array(masks),\n",
    "    np.array(image_with_masks),\n",
    ")\n",
    "width, height = image_with_masks.shape[1], image_with_masks.shape[0]\n",
    "\n",
    "fig = plt.figure(figsize=(20, 20))\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "ax = plot_bbox(image_with_masks, bboxes, scores, ids, class_names=labels, ax=ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd77b80c",
   "metadata": {},
   "source": [
    "### 3.5. Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a06e6fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "base_model_predictor.delete_model()\n",
    "base_model_predictor.delete_endpoint()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bba70fd4",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart_instance_segmentation|Amazon_JumpStart_Instance_Segmentation.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}