introduction_to_amazon_algorithms/jumpstart_image_embedding/Amazon_JumpStart_Image_Embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cad79bfa",
   "metadata": {},
   "source": [
    "# Introduction to JumpStart - Image Embedding"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6370cf3c",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c04b8a8a",
   "metadata": {},
   "source": [
    "---\n",
    "Welcome to Amazon [SageMaker JumpStart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html)! You can use JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through [SageMaker JumpStart API](https://sagemaker.readthedocs.io/en/stable/overview.html#use-prebuilt-models-with-sagemaker-jumpstart). \n",
    "\n",
    "In this demo notebook, we demonstrate how to use the JumpStart API for Image Embedding. Image Embedding is the task of converting an image into meaningful vector representations. We show how to use pre-trained model to find image embeddings. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1dc3a0e",
   "metadata": {},
   "source": [
    "1. [Set Up](#1.-Set-Up)\n",
    "2. [Select a model](#2.-Select-a-model)\n",
    "3. [Retrieve JumpStart Artifacts & Deploy an Endpoint](#3.-Retrieve-JumpStart-Artifacts-&-Deploy-an-Endpoint)\n",
    "4. [Download example images for inference](#4.-Download-example-images-for-inference)\n",
    "5. [Query endpoint and parse response](#5.-Query-endpoint-and-parse-response)\n",
    "6. [Clean up the endpoint](#6.-Clean-up-the-endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb85670d",
   "metadata": {},
   "source": [
    "Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43a535ba",
   "metadata": {},
   "source": [
    "### 1. Set Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89ba0717",
   "metadata": {},
   "source": [
    "---\n",
    "Before executing the notebook, there are some initial steps required for set up. This notebook requires latest version of sagemaker and ipywidgets\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f40034a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sagemaker ipywidgets --upgrade --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14bff8ef",
   "metadata": {},
   "source": [
    "#### Permissions and environment variables\n",
    "\n",
    "---\n",
    "To host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook as the AWS account role with SageMaker access. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bac7fed7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker, boto3, json\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "aws_role = get_execution_role()\n",
    "aws_region = boto3.Session().region_name\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7802fc14",
   "metadata": {},
   "source": [
    "### 2. Select a model\n",
    "\n",
    "***\n",
    "Here, we download jumpstart model_manifest file from the jumpstart s3 bucket, filter-out all the Image Embedding models and select a model for inference. \n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "179f6e8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import Dropdown\n",
    "\n",
    "# download JumpStart model_manifest file.\n",
    "boto3.client(\"s3\").download_file(\n",
    "    f\"jumpstart-cache-prod-{aws_region}\", \"models_manifest.json\", \"models_manifest.json\"\n",
    ")\n",
    "with open(\"models_manifest.json\", \"rb\") as json_file:\n",
    "    model_list = json.load(json_file)\n",
    "\n",
    "# filter-out all the Image Embedding models from the manifest list.\n",
    "image_embedding_models = []\n",
    "for model in model_list:\n",
    "    model_id = model[\"model_id\"]\n",
    "    if \"-icembedding-\" in model_id and model_id not in image_embedding_models:\n",
    "        image_embedding_models.append(model_id)\n",
    "\n",
    "# display the model-ids in a dropdown to select a model for inference.\n",
    "model_dropdown = Dropdown(\n",
    "    options=image_embedding_models,\n",
    "    value=\"tensorflow-icembedding-bit-m-r101x1-ilsvrc2012-featurevector-1\",\n",
    "    description=\"Select a model\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bda136d",
   "metadata": {},
   "source": [
    "#### Chose a model for Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec338a24",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(model_dropdown)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc30112c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_version=\"*\" fetches the latest version of the model\n",
    "model_id, model_version = model_dropdown.value, \"*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bab706dd",
   "metadata": {},
   "source": [
    "### 3. Retrieve JumpStart Artifacts & Deploy an Endpoint\n",
    "\n",
    "***\n",
    "\n",
    "We start by retrieving the `deploy_image_uri`, `deploy_source_uri`, and `model_uri` for the pre-trained model. To host the pre-trained model, we create an instance of [`sagemaker.model.Model`](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html) and deploy it. This may take a few minutes.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "341421cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import image_uris, model_uris, script_uris, hyperparameters\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "\n",
    "endpoint_name = name_from_base(f\"jumpstart-example-infer-{model_id}\")\n",
    "\n",
    "inference_instance_type = \"ml.p2.xlarge\"\n",
    "\n",
    "# Retrieve the inference docker container uri. This is the base Tensorflow container image for the default model above.\n",
    "deploy_image_uri = image_uris.retrieve(\n",
    "    region=None,\n",
    "    framework=None,  # automatically inferred from model_id\n",
    "    image_scope=\"inference\",\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    instance_type=inference_instance_type,\n",
    ")\n",
    "\n",
    "# Retrieve the inference script uri. This includes all dependencies and scripts for model loading, inference handling etc.\n",
    "deploy_source_uri = script_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, script_scope=\"inference\"\n",
    ")\n",
    "\n",
    "\n",
    "# Retrieve the model uri. This includes the model and model parameters.\n",
    "model_uri = model_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, model_scope=\"inference\"\n",
    ")\n",
    "\n",
    "\n",
    "# Create the SageMaker model instance\n",
    "model = Model(\n",
    "    image_uri=deploy_image_uri,\n",
    "    source_dir=deploy_source_uri,\n",
    "    model_data=model_uri,\n",
    "    entry_point=\"inference.py\",  # entry point file in source_dir and present in deploy_source_uri\n",
    "    role=aws_role,\n",
    "    predictor_cls=Predictor,\n",
    "    name=endpoint_name,\n",
    ")\n",
    "\n",
    "# deploy the Model. Note that we need to pass Predictor class when we deploy model through Model class,\n",
    "# for being able to run inference through the sagemaker API.\n",
    "model_predictor = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=inference_instance_type,\n",
    "    predictor_cls=Predictor,\n",
    "    endpoint_name=endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "264717bf",
   "metadata": {},
   "source": [
    "### 4. Download example images for inference\n",
    "---\n",
    "We download example images from the JumpStart assets S3 bucket.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9245da40",
   "metadata": {},
   "outputs": [],
   "source": [
    "region = boto3.Session().region_name\n",
    "s3_bucket = f\"jumpstart-cache-prod-{region}\"\n",
    "key_prefix = \"inference-notebook-assets\"\n",
    "s3 = boto3.client(\"s3\")\n",
    "\n",
    "\n",
    "def download_from_s3(key_filenames):\n",
    "    for key_filename in key_filenames:\n",
    "        s3.download_file(s3_bucket, f\"{key_prefix}/{key_filename}\", key_filename)\n",
    "\n",
    "\n",
    "cat_jpg, dog_jpg = \"cat.jpg\", \"dog.jpg\"\n",
    "download_from_s3(key_filenames=[cat_jpg, dog_jpg])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82a6c3d6",
   "metadata": {},
   "source": [
    "### 5. Query endpoint and parse response\n",
    "\n",
    "---\n",
    "Input to the endpoint is a single image in binary format. Output of the endpoint is a `json` with the image embedding.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c421e7a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def query(model_predictor, image_file_name):\n",
    "    \"\"\"Query the model predictor.\"\"\"\n",
    "\n",
    "    with open(image_file_name, \"rb\") as file:\n",
    "        input_img_rb = file.read()\n",
    "\n",
    "    query_response = model_predictor.predict(\n",
    "        input_img_rb,\n",
    "        {\n",
    "            \"ContentType\": \"application/x-image\",\n",
    "            \"Accept\": \"application/json\",\n",
    "        },\n",
    "    )\n",
    "    return query_response\n",
    "\n",
    "\n",
    "def parse_response(query_response):\n",
    "    \"\"\"Parse response and return the embedding.\"\"\"\n",
    "\n",
    "    model_predictions = json.loads(query_response)\n",
    "    translation_text = model_predictions[\"embedding\"]\n",
    "    return translation_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c53bd295",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import HTML\n",
    "\n",
    "for img_name in [cat_jpg, dog_jpg]:\n",
    "    query_response = query(model_predictor, img_name)\n",
    "    embedding = parse_response(query_response)\n",
    "    first_5element_embeddings = \"{\" + \", \".join([str(id) for id in embedding[:5]]) + \"}\"\n",
    "    display(\n",
    "        HTML(\n",
    "            f'<img src={img_name} alt={img_name} align=\"left\" style=\"width: 250px;\"/>'\n",
    "            f\"<figcaption>First-5 elements of the feature vector (embedding) are: {first_5element_embeddings}</figcaption>\"\n",
    "            f\"<figcaption>Total length of the feature vector (embedding) is: {len(embedding)}</figcaption>\"\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7494deef",
   "metadata": {},
   "source": [
    "## 6. Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdb0068b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "model_predictor.delete_model()\n",
    "model_predictor.delete_endpoint()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d9def101",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart_image_embedding|Amazon_JumpStart_Image_Embedding.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}