llama-2-lmi/llama-2-13b-chat/1-deploy-llama-2-13b-chat-lmi-response-streaming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a729a7fc-a1b6-4832-9155-d240ccd8ecc0",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Deploy LLama2 13B Chat LMI Model with response streaming on SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2958bcf-767f-4cd7-826e-963f91565876",
   "metadata": {},
   "source": [
    "\n",
    "In this notebook, we explore how to host a LLama2 13B Chat large language model on SageMaker using the DeepSpeed. We use DJLServing as the model serving solution in this example that is bundled in the Large Model Inference (LMI) container. DJLServing is a high-performance universal model serving solution powered by the Deep Java Library (DJL) that is programming language agnostic. To learn more about DJL and DJLServing, you can refer to our recent blog post (https://aws.amazon.com/blogs/machine-learning/deploy-bloom-176b-and-opt-30b-on-amazon-sagemaker-with-large-model-inference-deep-learning-containers-and-deepspeed/).\n",
    "\n",
    "\n",
    "Model parallelism can help deploy large models that would normally be too large for a single GPU. With model parallelism, we partition and distribute a model across multiple GPUs. Each GPU holds a different part of the model, resolving the memory capacity issue for the largest deep learning models with billions of parameters. \n",
    "\n",
    "SageMaker has rolled out DeepSpeed container which now provides users with the ability to leverage the managed serving capabilities and help to provide the un-differentiated heavy lifting.\n",
    "\n",
    "In this notebook, we deploy `'meta-llama/Llama-2-13b-chat-hf` model on a `ml.g5.12xlarge` instance. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0ae96c9-0255-45fa-bbb3-b5926df154c9",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Prerequisite\n",
    "### Hugging Face Account\n",
    "\n",
    "You need to have Hugging Face account. Sign Up here https://huggingface.co/join with your email if you do not already have account.\n",
    "\n",
    "- For seamless access of the models avaialble on Hugging Face especially gated models such as Llama, for fine-tuning and inferencing purposes, you need to have Hugging Face Account to obtain read Access Token.\n",
    "- After signup, [login](https://huggingface.co/login) to visit https://huggingface.co/settings/tokens to create read Access token.\n",
    "\n",
    "### Request access to the next version of Llama\n",
    "\n",
    "Use the same email id to obtain permission from meta by visiting this link: https://ai.meta.com/resources/models-and-libraries/llama-downloads/\n",
    "\n",
    "- The Llama models available via Hugging Face are gated models. The use of Llama model is governed by the Meta license. In order to download the model weights and tokenizer, please visit https://ai.meta.com/resources/models-and-libraries/llama-downloads/ and accept their License before requesting access.\n",
    "- Within 2 days you might be granted access to use Llama models via a confirmation email with subject: [Access granted] Your request to access model meta-llama/Llama-2-13b-chat-hf has been accepted. Though the model id is Llama-2-13b-chat-hf, you should be able to access other variants too.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dd6a36f-ee95-453d-a127-c8a7de6a026d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -Uq pip\n",
    "!pip install -Uq sagemaker boto3 huggingface_hub "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf0c89f4-679c-4557-b95d-1d954c15a020",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import jinja2\n",
    "from sagemaker import image_uris\n",
    "import boto3\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83d5a162-e9be-469b-910e-18cca8c359f8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "role = sagemaker.get_execution_role()  # execution role for the endpoint\n",
    "sess = sagemaker.session.Session()  # sagemaker session for interacting with different AWS APIs\n",
    "bucket = sess.default_bucket()  # bucket to house artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f68b5181-d018-4564-9762-fa8770a9672f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_prefix = \"hf-large-model-djl/meta-llama/Llama-2-13b-chat\"\n",
    "s3_code_prefix = f\"{s3_prefix}/code\"  # folder within bucket where code artifact will go\n",
    "s3_model_prefix = f\"{s3_prefix}/model\"  # folder within bucket where model artifact will go\n",
    "\n",
    "region = sess._region_name\n",
    "account_id = sess.account_id()\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "sm_client = boto3.client(\"sagemaker\")\n",
    "smr_client = boto3.client(\"sagemaker-runtime\")\n",
    "\n",
    "jinja_env = jinja2.Environment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "959e1413-76f8-4b01-88b5-1962c012d438",
   "metadata": {},
   "source": [
    "## Download the model snapshot from Hugging Face and upload the model artifacts on Amazon S3\n",
    "\n",
    "If you intend to download your copy of the model and upload it to a s3 location in your AWS account, please follow the below steps, else you can skip to the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be9d6e18-6760-4f88-835d-150ca372e6eb",
   "metadata": {},
   "source": [
    "Following Snapshot Download will take around 4 to 6 mins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94af859c-4c3a-4fda-ae27-890be565a906",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "from huggingface_hub import snapshot_download\n",
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "# - This will download the model into the current directory where ever the jupyter notebook is running\n",
    "local_model_path = Path(\".\")\n",
    "local_model_path.mkdir(exist_ok=True)\n",
    "model_name = 'meta-llama/Llama-2-13b-chat-hf'\n",
    "# Only download pytorch checkpoint files\n",
    "allow_patterns = [\"*.json\", \"*.txt\", \"*.model\", \"*.safetensors\", \"*.bin\", \"*.chk\", \"*.pth\"]\n",
    "\n",
    "# - Leverage the snapshot library to donload the model since the model is stored in repository using LFS\n",
    "model_download_path = snapshot_download(\n",
    "    repo_id=model_name, \n",
    "    cache_dir=local_model_path, \n",
    "    allow_patterns=allow_patterns, \n",
    "    token='<YOUR_HUGGING_FACE_READ_ACCESS_TOKEN>'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8716cd1-1170-4910-a5e9-34c194ac79d8",
   "metadata": {},
   "source": [
    "Upload files to default S3 bucket and obtain the URI in a variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f355f9f-69e2-4c1e-a467-5c5520a9b142",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_model_s3_uri = sess.upload_data(path=model_download_path, key_prefix=s3_model_prefix)\n",
    "print(f\"Model uploaded to --- > {base_model_s3_uri}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "993c024f-c18a-41e2-a784-90629732944e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Cleanup locally stored model files post S3 upload\n",
    "!rm -rf {model_download_path}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9913de41-193a-4bfb-b42e-581c7677d0ed",
   "metadata": {},
   "source": [
    "## Create SageMaker compatible Model artifact,  upload Model to S3 and bring your own inference script.\n",
    "\n",
    "SageMaker Large Model Inference containers can be used to host models without providing your own inference code. This is extremely useful when there is no custom pre-processing of the input data or postprocessing of the model's predictions.\n",
    "\n",
    "SageMaker needs the model artifacts to be in a Tarball format. In this example, we provide the following files - serving.properties.\n",
    "\n",
    "The tarball is in the following format:\n",
    "\n",
    "```\n",
    "code\n",
    "├──── \n",
    "│   └── serving.properties\n",
    "```\n",
    "\n",
    "    serving.properties is the configuration file that can be used to configure the model server.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b50f5d44-6388-49fa-bc87-2e82bed226f3",
   "metadata": {},
   "source": [
    "### Create serving.properties file for response streaming\n",
    "\n",
    "This is a configuration file to indicate to DJL Serving which model parallelization and inference optimization libraries you would like to use. Depending on your need, you can set the appropriate configuration.\n",
    "\n",
    "Here is a list of settings that we use in this configuration file -\n",
    "\n",
    "- `engine`: The runtime engine for DJL to use. The possible values for engine include *Python*, *DeepSpeed*, *FasterTransformer*, and *MPI*. In this case, we set it to MPI. MPI, Model Parallelization and Inference facilitates partitioning the model across all the available GPUs and thus accelerate the inference.\n",
    "- `option.tensor_parallel_degree` - This option specifies number of tensor parallel partitions performed on the model. Set to the number of GPU devices over which Accelerate needs to partition the model. This parameter also controls the no of workers per model which will be started up when DJL serving runs. For example if we have a 4 GPU machine and we are creating 4 partitions then we will have 1 worker per model to serve the requests.\n",
    "- `option.low_cpu_mem_usage` - Reduces CPU memory usage when loading models. We recommend that you set this to TRUE.\n",
    "- `option.rolling_batch` – Enables iteration-level batching using one of the supported strategies. Values include `auto`, `scheduler`, and `lmi-dist`. We use `lmi-dist` for turning on continuous batching for Llama 2.\n",
    "- `option.max_rolling_batch_size` – Limits the number of concurrent requests in the continuous batch. Defaults to 32.\n",
    "- `option.model_id`: The model id of a pretrained model hosted inside a [model repository on huggingface](https://huggingface.co/models) or S3 path to the model artefact. \n",
    "- `option.paged_attention` - Use PagedAttention or not. Default is always use. Disable this if you plan to run on G4 or older GPU architecture\n",
    "\n",
    "For more details on the configuration options and an exhaustive list, you can refer the documentation -\n",
    "\n",
    "[Model parallelism and large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-configuration.html)\n",
    "\n",
    "Since we are serving the model using deepspeed container, and Llama 2 being a large model used for inference,  we are following the approach of [Large model inference with DeepSpeed and DJL Serving](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-tutorials-deepspeed-djl.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2691f5b-4e98-4f7a-887c-49c05bbf7a8e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm -rf chat_llama2_13b_hf\n",
    "!mkdir -p chat_llama2_13b_hf\n",
    "model_id = base_model_s3_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6300047f-b377-4448-a446-2cd48121d103",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%writefile chat_llama2_13b_hf/serving.properties\n",
    "engine = MPI\n",
    "option.entryPoint=djl_python.huggingface\n",
    "option.tensor_parallel_degree=4\n",
    "option.low_cpu_mem_usage=TRUE\n",
    "option.rolling_batch=lmi-dist\n",
    "option.max_rolling_batch_size=64\n",
    "option.model_loading_timeout=900\n",
    "option.model_id={{model_id}}\n",
    "option.paged_attention=true"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bc7098f-9739-4504-a98f-553936b4f5ab",
   "metadata": {},
   "source": [
    "If you are using djl-deepspeed container version 0.25.0, you don't need to set the serving parameter `option.enable_streaming` to true, as the streaming is enabled by default. If you are using version 0.24.0, you will need to set `option.enable_streaming` to `true` for obtaining response stream while inferencing. Since we are deploying llama 2 13b, we are setting the **tensor_parallel_degree** to **4** and making use of all the 4 NVIDIA A10 GPUs available on the [`ml.g5.12xlarge`](https://aws.amazon.com/sagemaker/pricing/#:~:text=Amazon%20SageMaker%20G5%20instance%20product%20details) instance.\n",
    "\n",
    "https://aws.amazon.com/blogs/machine-learning/improve-throughput-performance-of-llama-2-models-using-amazon-sagemaker/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "996ebeb2-cfee-4e7b-af8f-3ccc811fa1eb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# we plug in the appropriate model location into our `serving.properties`\n",
    "template = jinja_env.from_string(Path(\"chat_llama2_13b_hf/serving.properties\").open().read())\n",
    "Path(\"chat_llama2_13b_hf/serving.properties\").open(\"w\").write(\n",
    "    template.render(model_id=base_model_s3_uri)\n",
    ")\n",
    "!pygmentize chat_llama2_13b_hf/serving.properties | cat -n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec9db9c4-5023-4125-a413-7c5afa135218",
   "metadata": {},
   "source": [
    "Image URI for the DJL container is being used here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dfabe0a-04f8-486d-94ab-7d6066680954",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "inference_image_uri = image_uris.retrieve(\n",
    "    framework=\"djl-deepspeed\", region=region, version=\"0.26.0\"\n",
    ")\n",
    "inference_image_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "905903de-a4b9-4f41-8cc2-564416ae5d5f",
   "metadata": {},
   "source": [
    "Create the Tarball and then upload to S3 location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c005aa2e-ec1a-4ccf-8f39-67bcbabd0309",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm model.tar.gz\n",
    "!tar czvf model.tar.gz chat_llama2_13b_hf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb83ba3b-2ea5-4297-8e85-f16dd4c7c13a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_code_artifact = sess.upload_data(\"model.tar.gz\", bucket, s3_code_prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c146caa-735b-433a-a673-89c74733dcb0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_code_artifact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dbbb5f3-c1e7-47ee-adcb-b9f8cc4add76",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm model.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6c9d3ba-c5a5-4605-9826-bf75dd3c42dc",
   "metadata": {},
   "source": [
    "## Deploy Llama 2 13B Chat LMI Model\n",
    "\n",
    "[Choosing instance types for large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-choosing-instance-types.html)\n",
    "\n",
    "We will proceed with deploying `meta-llama/Llama-2-13b-chat-hf` model on `ml.g5.12xlarge`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efa56b75-356a-4ebf-bebd-7150521c95e9",
   "metadata": {},
   "source": [
    "Steps to deploy the model to SageMaker Endpoint will be as follows:\n",
    "\n",
    "1. Create the Model using the Image container and the Model Tarball uploaded earlier\n",
    "2. Create the endpoint config using the following key parameters\n",
    "\n",
    "    a) Instance Type is ml.g5.12xlarge \n",
    "    \n",
    "    b) ContainerStartupHealthCheckTimeoutInSeconds is 3600 to ensure health check starts after the model is ready    \n",
    "3. Create the end point using the endpoint config created    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddee8273-8009-4739-af67-79525b6882cd",
   "metadata": {},
   "source": [
    "#### Create the Model\n",
    "Use the image URI for the DJL container and the s3 location to which the tarball was uploaded.\n",
    "\n",
    "The container downloads the model into the `/tmp` space on the instance because SageMaker maps the `/tmp` to the Amazon Elastic Block Store (Amazon EBS) volume that is mounted when we specify the endpoint creation parameter VolumeSizeInGB. \n",
    "It leverages `s5cmd`(https://github.com/peak/s5cmd) which offers a very fast download speed and hence extremely useful when downloading large models.\n",
    "\n",
    "For instances like p4dn, which come pre-built with the volume instance, we can continue to leverage the `/tmp` on the container. The size of this mount is large enough to hold the model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eac40dc4-c9c3-4af1-8d54-7b6742259266",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "model_name = name_from_base(f\"Llama-2-13b-chat-lmi-streaming\")\n",
    "\n",
    "create_model_response = sm_client.create_model(\n",
    "    ModelName=model_name,\n",
    "    ExecutionRoleArn=role,\n",
    "    PrimaryContainer={\n",
    "        \"Image\": inference_image_uri,\n",
    "        \"ModelDataUrl\": s3_code_artifact,\n",
    "        \"Environment\": {\"MODEL_LOADING_TIMEOUT\": \"3600\"},\n",
    "    },\n",
    ")\n",
    "model_arn = create_model_response[\"ModelArn\"]\n",
    "\n",
    "#print(f\"Created Model: {model_arn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "891cbe78-a4a6-4387-8c2b-3bcd5c910deb",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Create the Endpoint Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "332e4a25-233f-4e75-aabb-0986c6c48f77",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "endpoint_config_name = f\"{model_name}-config\"\n",
    "\n",
    "endpoint_name = name_from_base(model_name)\n",
    "\n",
    "endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"variant1\",\n",
    "            \"ModelName\": model_name,\n",
    "            \"InstanceType\": \"ml.g5.12xlarge\",\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            \"ModelDataDownloadTimeoutInSeconds\": 3600,\n",
    "            \"ContainerStartupHealthCheckTimeoutInSeconds\": 3600,\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "#endpoint_config_response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37b14b8c-4002-40c2-b157-8b6997060c62",
   "metadata": {},
   "source": [
    "#### Create Endpoint and Deploy the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b6a0ca5-3ee6-4793-83f4-d9565a7d06e5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=f\"{endpoint_name}\", EndpointConfigName=endpoint_config_name\n",
    ")\n",
    "#print(f\"Created Endpoint: {create_endpoint_response['EndpointArn']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f893ecb0-e0fe-450a-8bb1-5709d07b6e9e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import time\n",
    "\n",
    "resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = resp[\"EndpointStatus\"]\n",
    "print(\"Status: \" + status)\n",
    "\n",
    "while status == \"Creating\":\n",
    "    time.sleep(60)\n",
    "    resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp[\"EndpointStatus\"]\n",
    "    print(\"🌊\", end='')\n",
    "\n",
    "if status ==\"InService\":\n",
    "    print('🏄‍♂️')\n",
    "else:\n",
    "    print('💦')\n",
    "print(\"\\nArn: \" + resp[\"EndpointArn\"])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d0ec2d7-6ef8-4383-ac50-6a4984af5c03",
   "metadata": {},
   "source": [
    "#### While you wait for the endpoint to be created, you can read more about:\n",
    "- [Deep Learning containers for large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-large-model-dlc.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68aed031-b68c-4e23-8fa7-24483ec189d1",
   "metadata": {},
   "source": [
    "We will store the value of the variable endpoint_name to use it in inference notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82bca632-7305-4dcb-bf96-ae76337ab53a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%store \\\n",
    "endpoint_name \\\n",
    "bucket \\\n",
    "s3_prefix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6a3dec0-6c74-449a-a614-0ac0c82433fc",
   "metadata": {},
   "source": [
    "## References:\n",
    "\n",
    "- [Improve throughput performance of Llama 2 models using Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/improve-throughput-performance-of-llama-2-models-using-amazon-sagemaker/)\n",
    "- [Improve performance of Falcon models with Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/improve-performance-of-falcon-models-with-amazon-sagemaker/)\n",
    "- [serving.properties - Configurations and settings](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-configuration.html)\n",
    "- [Amazon SageMaker launches a new version of Large Model Inference DLC with TensorRT-LLM support](https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-sagemaker-large-model-inference-dlc-tensorrt-llm-support/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55fa97ed-f52f-4456-a214-ca2302f2844f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.r5.4xlarge",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}