05b - Convolutional Neural Networks (Tensorflow).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Networks with TensorFlow\n",
    "\n",
    "\"Deep Learning\" is a general term that usually refers to the use of neural networks with multiple layers that synthesize the way the human brain learns and makes decisions. A convolutional neural network is a kind of neural network that extracts *features* from matrices of numeric values (often images) by convolving multiple filters over the matrix values to apply weights and identify patterns, such as edges, corners, and so on in an image. The numeric representations of these patterns are then passed to a fully-connected neural network layer to map the features to specific classes.\n",
    "\n",
    "There are several commonly used frameworks for creating CNNs. In this notebook, we'll build a simple example CNN using TensorFlow.\n",
    "\n",
    "## Install and import libraries\n",
    "\n",
    "First, let's install and import the TensorFlow libraries we'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "from  tensorflow import keras\n",
    "print('TensorFlow version:',tensorflow.__version__)\n",
    "print('Keras version:',keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the data\n",
    "\n",
    "In this exercise, you'll train a CNN-based classification model that can classify images of geometric shapes. Let's take a look at the classes of shape the model needs to identify."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import os\n",
    "%matplotlib inline\n",
    "\n",
    "# The images are in the data/shapes folder\n",
    "data_folder = 'data/shapes'\n",
    "\n",
    "# Get the class names\n",
    "classes = os.listdir(data_folder)\n",
    "classes.sort()\n",
    "print(len(classes), 'classes:')\n",
    "print(classes)\n",
    "\n",
    "# Show the first image in each folder\n",
    "fig = plt.figure(figsize=(8, 12))\n",
    "i = 0\n",
    "for sub_dir in os.listdir(data_folder):\n",
    "    i+=1\n",
    "    img_file = os.listdir(os.path.join(data_folder,sub_dir))[0]\n",
    "    img_path = os.path.join(data_folder, sub_dir, img_file)\n",
    "    img = mpimg.imread(img_path)\n",
    "    a=fig.add_subplot(1, len(classes),i)\n",
    "    a.axis('off')\n",
    "    imgplot = plt.imshow(img)\n",
    "    a.set_title(img_file)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the data\n",
    "\n",
    "Before we can train the model, we need to prepare the data. We'll divide the feature values by 255 to normalize them as floating point values between 0 and 1, and we'll split the data so that we can use 70% of it to train the model, and hold back 30% to validate it. When loading the data, the data generator will assign \"hot-encoded\" numeric labels to indicate which class each image belongs to based on the subfolders in which the data is stored. In this case, there are three subfolders - *circle*, *square*, and *triangle*, so the labels will consist of three *0* or *1* values indicating which of these classes is associated with the image - for example the label [0 1 0] indicates that the image belongs to the second class (*square*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "img_size = (128, 128)\n",
    "batch_size = 30\n",
    "\n",
    "print(\"Getting Data...\")\n",
    "datagen = ImageDataGenerator(rescale=1./255, # normalize pixel values\n",
    "                             validation_split=0.3) # hold back 30% of the images for validation\n",
    "\n",
    "print(\"Preparing training dataset...\")\n",
    "train_generator = datagen.flow_from_directory(\n",
    "    data_folder,\n",
    "    target_size=img_size,\n",
    "    batch_size=batch_size,\n",
    "    class_mode='categorical',\n",
    "    subset='training') # set as training data\n",
    "\n",
    "print(\"Preparing validation dataset...\")\n",
    "validation_generator = datagen.flow_from_directory(\n",
    "    data_folder,\n",
    "    target_size=img_size,\n",
    "    batch_size=batch_size,\n",
    "    class_mode='categorical',\n",
    "    subset='validation') # set as validation data\n",
    "\n",
    "classnames = list(train_generator.class_indices.keys())\n",
    "print('Data generators ready')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the CNN\n",
    "\n",
    "Now we're ready to create our model. This involves defining the layers for our CNN, and compiling them for multi-class classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a CNN classifier network\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense\n",
    "\n",
    "# Define the model as a sequence of layers\n",
    "model = Sequential()\n",
    "\n",
    "# The input layer accepts an image and applies a convolution that uses 32 6x6 filters and a rectified linear unit activation function\n",
    "model.add(Conv2D(32, (6, 6), input_shape=train_generator.image_shape, activation='relu'))\n",
    "\n",
    "# Next we'll add a max pooling layer with a 2x2 patch\n",
    "model.add(MaxPooling2D(pool_size=(2,2)))\n",
    "\n",
    "# We can add as many layers as we think necessary - here we'll add another convolution and max pooling layer\n",
    "model.add(Conv2D(32, (6, 6), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "# And another set\n",
    "model.add(Conv2D(32, (6, 6), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "# A dropout layer randomly drops some nodes to reduce inter-dependencies (which can cause over-fitting)\n",
    "model.add(Dropout(0.2))\n",
    "\n",
    "# Flatten the feature maps \n",
    "model.add(Flatten())\n",
    "\n",
    "# Generate a fully-connected output layer with a predicted probability for each class\n",
    "# (softmax ensures all probabilities sum to 1)\n",
    "model.add(Dense(train_generator.num_classes, activation='softmax'))\n",
    "\n",
    "# With the layers defined, we can now compile the model for categorical (multi-class) classification\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "\n",
    "With the layers of the CNN defined, we're ready to train the model using our image data. In the example below, we use 5 iterations (*epochs*) to train the model in 30-image batches, holding back 30% of the data for validation. After each epoch, the loss function measures the error (*loss*) in the model and adjusts the weights (which were randomly generated for the first iteration) to try to improve accuracy. \n",
    "\n",
    "> **Note**: We're only using 5 epochs to minimize the training time for this simple example. A real-world CNN is usually trained over more epochs than this. CNN model training is processor-intensive, involving a lot of matrix and vector-based operations; so it's recommended to perform this on a system that can leverage GPUs, which are optimized for these kinds of calculation. This will take a while to complete on a CPU-based system - status will be displayed as the training progresses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Train the model over 5 epochs using 30-image batches and using the validation holdout dataset for validation\n",
    "num_epochs = 5\n",
    "history = model.fit(\n",
    "    train_generator,\n",
    "    steps_per_epoch = train_generator.samples // batch_size,\n",
    "    validation_data = validation_generator, \n",
    "    validation_steps = validation_generator.samples // batch_size,\n",
    "    epochs = num_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the loss history\n",
    "\n",
    "We tracked average training and validation loss history for each epoch. We can plot these to verify that loss reduced as the model was trained, and to detect *overfitting* (which is indicated by a continued drop in training loss after validation loss has levelled out or started to increase)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "epoch_nums = range(1,num_epochs+1)\n",
    "training_loss = history.history[\"loss\"]\n",
    "validation_loss = history.history[\"val_loss\"]\n",
    "plt.plot(epoch_nums, training_loss)\n",
    "plt.plot(epoch_nums, validation_loss)\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('loss')\n",
    "plt.legend(['training', 'validation'], loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate model performance\n",
    "\n",
    "We can see the final accuracy based on the test data, but typically we'll want to explore performance metrics in a little more depth. Let's plot a confusion matrix to see how well the model is predicting each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Tensorflow doesn't have a built-in confusion matrix metric, so we'll use SciKit-Learn\n",
    "import numpy as np\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "print(\"Generating predictions from validation data...\")\n",
    "# Get the image and label arrays for the first batch of validation data\n",
    "x_test = validation_generator[0][0]\n",
    "y_test = validation_generator[0][1]\n",
    "\n",
    "# Use the model to predict the class\n",
    "class_probabilities = model.predict(x_test)\n",
    "\n",
    "# The model returns a probability value for each class\n",
    "# The one with the highest probability is the predicted class\n",
    "predictions = np.argmax(class_probabilities, axis=1)\n",
    "\n",
    "# The actual labels are hot encoded (e.g. [0 1 0], so get the one with the value 1\n",
    "true_labels = np.argmax(y_test, axis=1)\n",
    "\n",
    "# Plot the confusion matrix\n",
    "cm = confusion_matrix(true_labels, predictions)\n",
    "plt.imshow(cm, interpolation=\"nearest\", cmap=plt.cm.Blues)\n",
    "plt.colorbar()\n",
    "tick_marks = np.arange(len(classnames))\n",
    "plt.xticks(tick_marks, classnames, rotation=85)\n",
    "plt.yticks(tick_marks, classnames)\n",
    "plt.xlabel(\"Predicted Shape\")\n",
    "plt.ylabel(\"Actual Shape\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the Trained model\n",
    "\n",
    "Now that you've trained a working model, you can save it (including the trained weights) for use later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the trained model\n",
    "modelFileName = 'models/shape_classifier.h5'\n",
    "model.save(modelFileName)\n",
    "del model  # deletes the existing model variable\n",
    "print('model saved as', modelFileName)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the trained model\n",
    "\n",
    "When you have a new image, you can use the saved model to predict its class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import models\n",
    "import numpy as np\n",
    "from random import randint\n",
    "import os\n",
    "%matplotlib inline\n",
    "\n",
    "# Function to predict the class of an image\n",
    "def predict_image(classifier, image):\n",
    "    from tensorflow import convert_to_tensor\n",
    "    # The model expects a batch of images as input, so we'll create an array of 1 image\n",
    "    imgfeatures = image.reshape(1, image.shape[0], image.shape[1], image.shape[2])\n",
    "\n",
    "    # We need to format the input to match the training data\n",
    "    # The generator loaded the values as floating point numbers\n",
    "    # and normalized the pixel values, so...\n",
    "    imgfeatures = imgfeatures.astype('float32')\n",
    "    imgfeatures /= 255\n",
    "    \n",
    "    # Use the model to predict the image class\n",
    "    class_probabilities = classifier.predict(imgfeatures)\n",
    "    \n",
    "    # Find the class predictions with the highest predicted probability\n",
    "    index = int(np.argmax(class_probabilities, axis=1)[0])\n",
    "    return index\n",
    "\n",
    "# Function to create a random image (of a square, circle, or triangle)\n",
    "def create_image (size, shape):\n",
    "    from random import randint\n",
    "    import numpy as np\n",
    "    from PIL import Image, ImageDraw\n",
    "    \n",
    "    xy1 = randint(10,40)\n",
    "    xy2 = randint(60,100)\n",
    "    col = (randint(0,200), randint(0,200), randint(0,200))\n",
    "\n",
    "    img = Image.new(\"RGB\", size, (255, 255, 255))\n",
    "    draw = ImageDraw.Draw(img)\n",
    "    \n",
    "    if shape == 'circle':\n",
    "        draw.ellipse([(xy1,xy1), (xy2,xy2)], fill=col)\n",
    "    elif shape == 'triangle':\n",
    "        draw.polygon([(xy1,xy1), (xy2,xy2), (xy2,xy1)], fill=col)\n",
    "    else: # square\n",
    "        draw.rectangle([(xy1,xy1), (xy2,xy2)], fill=col)\n",
    "    del draw\n",
    "    \n",
    "    return np.array(img)\n",
    "\n",
    "# Create a random test image\n",
    "classnames = os.listdir(os.path.join('data', 'shapes'))\n",
    "classnames.sort()\n",
    "img = create_image ((128,128), classnames[randint(0, len(classnames)-1)])\n",
    "plt.axis('off')\n",
    "plt.imshow(img)\n",
    "\n",
    "# Use the classifier to predict the class\n",
    "model = models.load_model(modelFileName) # loads the saved model\n",
    "class_idx = predict_image(model, img)\n",
    "print (classnames[class_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Reading\n",
    "\n",
    "To learn more about training convolutional neural networks with TensorFlow, see the [TensorFlow documentation](https://www.tensorflow.org/overview).\n",
    "\n",
    "## Challenge: Safari Image Classification\n",
    "\n",
    "Hopefully this notebook has shown you the main steps in training and evaluating a CNN. Why not put what you've learned into practice with our Safari image classification challenge in the [/challenges/05 - Safari CNN Challenge.ipynb](./challenges/05%20-%20Safari%20CNN%20Challenge.ipynb) notebook?\n",
    "\n",
    "> **Note**: The time to complete this optional challenge is not included in the estimated time for this exercise - you can spend as little or as much time on it as you like!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}