sagemaker-debugger/tensorflow_builtin_rule/tf-mnist-builtin-rule.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89862d8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install smdebug matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90320f6c",
   "metadata": {
    "papermill": {
     "duration": 0.015745,
     "end_time": "2021-06-01T00:12:47.822109",
     "exception": false,
     "start_time": "2021-06-01T00:12:47.806364",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Amazon SageMaker Debugger Tutorial: How to Use the Built-in Debugging Rules"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d972795c",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "255858c1",
   "metadata": {
    "papermill": {
     "duration": 0.015761,
     "end_time": "2021-06-01T00:12:47.853783",
     "exception": false,
     "start_time": "2021-06-01T00:12:47.838022",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "[Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) is a feature that offers capability to debug training jobs of your machine learning model and identify training problems in real time. While a training job looks like it's working like a charm, the model might have some common problems, such as loss not decreasing, overfitting, and underfitting. To better understand, practitioners have to debug the training job, while it can be challenging to track and analyze all of the output tensors.\n",
    "\n",
    "SageMaker Debugger covers the major deep learning frameworks (TensorFlow, PyTorch, and MXNet) and machine learning algorithm (XGBoost) to do the debugging jobs with minimal coding. Debugger provides an automatic detection of training problems through its built-in rules, and you can find a full list of the built-in rules for debugging at [List of Debugger Built-in Rules](https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-built-in-rules.html). \n",
    "\n",
    "In this tutorial, you will learn how to use SageMaker Debugger and its built-in rules to debug your model.\n",
    "\n",
    "The workflow is as follows:\n",
    "* [Step 1: Import SageMaker Python SDK and the Debugger client library smdebug](#step1)\n",
    "* [Step 2: Create a Debugger built-in rule list object](#step2)\n",
    "* [Step 3: Construct a SageMaker estimator](#step3)\n",
    "* [Step 4: Run the training job](#step4)\n",
    "* [Step 5: Check training progress on Studio Debugger insights dashboard and the built-in rules evaluation status](#step5)\n",
    "* [Step 6: Create a Debugger trial object to access the saved tensors](#step6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "409d1ad5",
   "metadata": {
    "papermill": {
     "duration": 0.015708,
     "end_time": "2021-06-01T00:12:47.885275",
     "exception": false,
     "start_time": "2021-06-01T00:12:47.869567",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"step2\"></a>\n",
    "## Step 1: Import SageMaker Python SDK and the SMDebug client library"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c3c36c2",
   "metadata": {
    "papermill": {
     "duration": 0.015697,
     "end_time": "2021-06-01T00:12:47.916797",
     "exception": false,
     "start_time": "2021-06-01T00:12:47.901100",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<font color='red'>**Important**</font>: To use the new Debugger features, you need to upgrade the SageMaker Python SDK and the SMDebug libary. In the following cell, change the third line to `install_needed=True` and run to upgrade the libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d67e6eb6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-06-01T00:12:47.954649Z",
     "iopub.status.busy": "2021-06-01T00:12:47.954138Z",
     "iopub.status.idle": "2021-06-01T00:12:47.955877Z",
     "shell.execute_reply": "2021-06-01T00:12:47.956215Z"
    },
    "papermill": {
     "duration": 0.023813,
     "end_time": "2021-06-01T00:12:47.956349",
     "exception": false,
     "start_time": "2021-06-01T00:12:47.932536",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import IPython\n",
    "install_needed = False  # Set to True to upgrade\n",
    "if install_needed:\n",
    "    print(\"installing deps and restarting kernel\")\n",
    "    !{sys.executable} -m pip install -U sagemaker\n",
    "    !{sys.executable} -m pip install -U smdebug\n",
    "    IPython.Application.instance().kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae58abe9",
   "metadata": {
    "papermill": {
     "duration": 0.015886,
     "end_time": "2021-06-01T00:12:47.988259",
     "exception": false,
     "start_time": "2021-06-01T00:12:47.972373",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Check the SageMaker Python SDK and the SMDebug library versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "207efdee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-06-01T00:12:48.024143Z",
     "iopub.status.busy": "2021-06-01T00:12:48.023410Z",
     "iopub.status.idle": "2021-06-01T00:12:48.473319Z",
     "shell.execute_reply": "2021-06-01T00:12:48.473695Z"
    },
    "papermill": {
     "duration": 0.469517,
     "end_time": "2021-06-01T00:12:48.473846",
     "exception": false,
     "start_time": "2021-06-01T00:12:48.004329",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "sagemaker.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8ca64fe",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-06-01T00:12:48.510517Z",
     "iopub.status.busy": "2021-06-01T00:12:48.509765Z",
     "iopub.status.idle": "2021-06-01T00:12:48.592483Z",
     "shell.execute_reply": "2021-06-01T00:12:48.591609Z"
    },
    "papermill": {
     "duration": 0.102182,
     "end_time": "2021-06-01T00:12:48.592688",
     "exception": true,
     "start_time": "2021-06-01T00:12:48.490506",
     "status": "failed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import smdebug\n",
    "\n",
    "smdebug.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0bf4e9d",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"step1\"></a>\n",
    "## Step 2: Create a Debugger built-in rule list object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae2b47ea",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.debugger import Rule, ProfilerRule, rule_configs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0dfd2d5",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "The following code cell shows how to configure a rule object for debugging and profiling. For more information about the Debugger built-in rules, see [List of Debugger Built-in Rules](https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-built-in-rules.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7daa52d1",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "built_in_rules = [\n",
    "    Rule.sagemaker(rule_configs.overfit()),\n",
    "    ProfilerRule.sagemaker(rule_configs.ProfilerReport()),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1ca378d",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"step3\"></a>\n",
    "## Step 3: Construct a SageMaker estimator\n",
    "\n",
    "Using the rule object created in the previous cell, construct a SageMaker estimator. \n",
    "\n",
    "The estimator can be one of the SageMaker framework estimators, [TensorFlow](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-estimator), [PyTorch](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html), [MXNet](https://sagemaker.readthedocs.io/en/stable/frameworks/mxnet/sagemaker.mxnet.html#mxnet-estimator), and [XGBoost](https://sagemaker.readthedocs.io/en/stable/frameworks/xgboost/xgboost.html), or the [SageMaker generic estimator](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html#sagemaker.estimator.Estimator). For more information about what framework versions are supported, see [Debugger-supported Frameworks and Algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html#debugger-supported-aws-containers).\n",
    "\n",
    "In this tutorial, the SageMaker TensorFlow estimator is constructed to run a TensorFlow training script with the Keras ResNet50 model and the cifar10 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "416e40ae",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "session = boto3.session.Session()\n",
    "region = session.region_name\n",
    "\n",
    "estimator = TensorFlow(\n",
    "    role=sagemaker.get_execution_role(),\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.p3.8xlarge\",\n",
    "    image_uri=f\"763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-training:2.3.1-gpu-py37-cu110-ubuntu18.04\",\n",
    "    # framework_version='2.3.1',\n",
    "    # py_version=\"py37\",\n",
    "    max_run=3600,\n",
    "    source_dir=\"./src\",\n",
    "    entry_point=\"tf-resnet50-cifar10.py\",\n",
    "    # Debugger Parameters\n",
    "    rules=built_in_rules,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3d69eae",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"step4\"></a>\n",
    "## Step 4: Run the training job\n",
    "With the `wait=False` option, you can proceed to the next notebook cell without waiting for the training job logs to be printed out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94f579db",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator.fit(wait=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23be49b5",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"step5\"></a>\n",
    "## Step 5: Check training progress on Studio Debugger insights dashboard and the built-in rules evaluation status\n",
    "\n",
    "- **Option 1** - Use SageMaker Studio Debugger insights and Experiments. This is a non-coding approach.\n",
    "- **Option 2** - Use the following code cells. This is a code-based approach. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fc652dd",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### Option 1 - Open Studio Debugger insights dashboard to get insights into the training job\n",
    "\n",
    "Through the Debugger insights dashboard on Studio, you can check the training jobs status, system resource utilization, and suggestions to optimize model performance. The following screenshot shows the Debugger insights dashboard interface.\n",
    "\n",
    "<IMG src=images/studio-debugger-insights-dashboard.png/>\n",
    "\n",
    "The following heatmap shows the `ml.p3.8xlarge` instance utilization while the training job is running or after the job has completed. To learn how to access the Debugger insights dashboard, see [Debugger on Studio](https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-on-studio.html) in the [SageMaker Debugger developer guide](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html).\n",
    "\n",
    "<IMG src=images/studio-debugger-insights-heatmap.png/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1dd554c",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### Option 2 - Run the following scripts for the code-based option\n",
    "\n",
    "The following two code cells return the current training job name, status, and the rule status in real time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84c01d95",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "##### Print the training job name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86957306",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "job_name = estimator.latest_training_job.name\n",
    "print(\"Training job name: {}\".format(job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad6d8918",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "##### Print the training job and rule evaluation status\n",
    "\n",
    "The following script returns the status in real time every 15 seconds, until the secondary training status turns to one of the descriptions, `Training`, `Stopped`, `Completed`, or `Failed`. Once the training job status turns into the `Training`, you will be able to retrieve tensors from the default S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b078d590",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "client = estimator.sagemaker_session.sagemaker_client\n",
    "description = client.describe_training_job(TrainingJobName=job_name)\n",
    "if description[\"TrainingJobStatus\"] != \"Completed\":\n",
    "    while description[\"SecondaryStatus\"] not in {\"Training\", \"Stopped\", \"Completed\", \"Failed\"}:\n",
    "        description = client.describe_training_job(TrainingJobName=job_name)\n",
    "        primary_status = description[\"TrainingJobStatus\"]\n",
    "        secondary_status = description[\"SecondaryStatus\"]\n",
    "        print(\n",
    "            \"Current job status: [PrimaryStatus: {}, SecondaryStatus: {}] | {} Rule Evaluation Status: {}\".format(\n",
    "                primary_status,\n",
    "                secondary_status,\n",
    "                estimator.latest_training_job.rule_job_summary()[0][\"RuleConfigurationName\"],\n",
    "                estimator.latest_training_job.rule_job_summary()[0][\"RuleEvaluationStatus\"],\n",
    "            )\n",
    "        )\n",
    "        time.sleep(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0123536",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"step6\"></a>\n",
    "## Step 6: Create a Debugger trial object to access the saved model parameters\n",
    "\n",
    "To access the saved tensors by Debugger, use the `smdebug` client library to create a Debugger trial object. The following code cell sets up a `tutorial_trial` object, and waits until it finds available tensors from the default S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70f1f431",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from smdebug.trials import create_trial\n",
    "\n",
    "tutorial_trial = create_trial(estimator.latest_job_debugger_artifacts_path())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "902f9530",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "The Debugger trial object accesses the SageMaker estimator's Debugger artifact path, and fetches the output tensors saved for debugging."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad8b7975",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### Print the default S3 bucket URI where the Debugger output tensors are stored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58fc0a76",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "tutorial_trial.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d14da2f",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### Print the Debugger output tensor names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbda4e66",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "tutorial_trial.tensor_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed1b1a9f",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### Print the list of steps where the tensors are saved"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d417a023",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "The smdebug `ModeKeys` class provides training phase mode keys that you can use to sort training (`TRAIN`) and validation (`EVAL`) steps and their corresponding values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cefeaaa",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from smdebug.core.modes import ModeKeys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42ebaf93",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "tutorial_trial.steps(mode=ModeKeys.TRAIN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3d48e6a",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "tutorial_trial.steps(mode=ModeKeys.EVAL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c97edd1e",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### Plot the loss curve\n",
    "\n",
    "The following script plots the loss and accuracy curves of training and validation loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "525d3b84",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "trial = tutorial_trial\n",
    "\n",
    "\n",
    "def get_data(trial, tname, mode):\n",
    "    tensor = trial.tensor(tname)\n",
    "    steps = tensor.steps(mode=mode)\n",
    "    vals = [tensor.value(s, mode=mode) for s in steps]\n",
    "    return steps, vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1537e9ea",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.axes_grid1 import host_subplot\n",
    "\n",
    "\n",
    "def plot_tensor(trial, tensor_name):\n",
    "    tensor_name = tensor_name\n",
    "    steps_train, vals_train = get_data(trial, tensor_name, mode=ModeKeys.TRAIN)\n",
    "    steps_eval, vals_eval = get_data(trial, tensor_name, mode=ModeKeys.EVAL)\n",
    "\n",
    "    fig = plt.figure(figsize=(10, 7))\n",
    "    host = host_subplot(111)\n",
    "\n",
    "    par = host.twiny()\n",
    "\n",
    "    host.set_xlabel(\"Steps (TRAIN)\")\n",
    "    par.set_xlabel(\"Steps (EVAL)\")\n",
    "    host.set_ylabel(tensor_name)\n",
    "\n",
    "    (p1,) = host.plot(steps_train, vals_train, label=tensor_name)\n",
    "    (p2,) = par.plot(steps_eval, vals_eval, label=\"val_\" + tensor_name)\n",
    "\n",
    "    leg = plt.legend()\n",
    "\n",
    "    host.xaxis.get_label().set_color(p1.get_color())\n",
    "    leg.texts[0].set_color(p1.get_color())\n",
    "\n",
    "    par.xaxis.get_label().set_color(p2.get_color())\n",
    "    leg.texts[1].set_color(p2.get_color())\n",
    "\n",
    "    plt.ylabel(tensor_name)\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_tensor(trial, \"loss\")\n",
    "plot_tensor(trial, \"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6554a41",
   "metadata": {},
   "source": [
    "> ## Note : Rerun the above cell if you don't see any plots! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "990c6773",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "In this tutorial, you learned how to use SageMaker Debugger with the minimal coding through SageMaker Studio and Jupyter notebook. The Debugger built-in rules detect training anomalies while concurrently reading in the output tensors, such as weights, activation outputs, gradients, accuracy, and loss, from your training jobs. In the next tutorial videos, you will learn more features of Debugger, such as how to analyze the tensors, change the built-in debugging rule parameters and thresholds, and save the tensors at your preferred S3 bucket URI."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8f11d2c3",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/sagemaker-debugger|tensorflow_builtin_rule|tf-mnist-builtin-rule.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (TensorFlow 2.10.0 Python 3.9 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/tensorflow-2.10.1-cpu-py39-ubuntu20.04-sagemaker-v1.2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 3.063866,
   "end_time": "2021-06-01T00:12:50.004015",
   "environment_variables": {},
   "exception": true,
   "input_path": "tf-mnist-builtin-rule.ipynb",
   "output_path": "/opt/ml/processing/output/tf-mnist-builtin-rule-2021-06-01-00-09-02.ipynb",
   "parameters": {
    "kms_key": "arn:aws:kms:us-west-2:521695447989:key/6e9984db-50cf-4c7e-926c-877ec47a8b25"
   },
   "start_time": "2021-06-01T00:12:46.940149",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}