notebooks/permuted_and_split_mnist.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "permuted_and_split_mnist.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.1"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fp9hbokQIxWW"
      },
      "source": [
        "# Permuted and Split MNIST: a Deep Continual Learning Example in PyTorch\n",
        "\n",
        "In this brief demo we will showcase two common *Continual Learning* benchmark often used to introduce the problem and start prototyping possible computational strategies to solve it. We will use bare Python, Numpy and *PyTorch*. In order to construct these benchmarks we will start from the the standard MNIST dataset (LeCun, 1998)!\n",
        "\n",
        "This notebook is part of the **[Continual AI Colab](https://github.com/ContinualAI/colab)**, a repository meant for tutorials and demo running on Google Colaboratory. [Continual AI](https://www.continualai.org/) is an open research community on the topic of Continual Learning and AI! Join us today [on slack](https://continualai.herokuapp.com/)! :-D\n",
        "\n",
        "This notebook has been also used for the [Continual Learning course](https://course.continualai.org) offered at the *University of Pisa* in conjunction with *ContinualAI* and the *AIDA Doctoral Academy*.\n",
        "\n",
        "We will start with learning over the standard *MNIST* benchmark, then we will move in the actual continual learning setting  with the *Permuted MNIST* and *Split MNIST*  benchmarks. Let's have some fun! :-)\n",
        "\n",
        "\n",
        "---\n",
        "**Connecting a local runtime**\n",
        "\n",
        "In case resources are not enough for you (no GPU for example), you can always connect another [local runtime](https://research.google.com/colaboratory/local-runtimes.html) or to a [runtime on a Google Compute Engine instance](https://research.google.com/colaboratory/local-runtimes.html).\n",
        "\n",
        "This notebook has been designed to run fast enough on simple CPUs so you shouldn't find any trouble here, using a free *hosted account*.\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "**Requisites to run it locally, outside colab (not recommended)**\n",
        "\n",
        "*   Python 3.x\n",
        "*   Jupyter\n",
        "*   PyTorch >= 1.8\n",
        "*   NumPy\n",
        "*   Matplotlib\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z6RUp96FLuMd"
      },
      "source": [
        "# Google Colaboratory\n",
        "\n",
        "First of all, take a moment to look around and discover Google Colab if you haven't before! You can run the commands below to understand how much resources you're using and are still available. Then consider also that you can also connect your Google Drive for additional space or for easily loading your own files.\n",
        "\n",
        "You can always reset the entire VM with \"*Runtime > Reset all runtime*\" in case of difficulty. Make also sure you're using the GPU or TPU in the same  tab (\"*Runtime > Change runtime type*\")."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pPViRmMBqbJ2",
        "outputId": "da8e256d-9a74-43c4-aae6-a4ee45d43ab0"
      },
      "source": [
        "!free -m\n",
        "!df -h\n",
        "!nvidia-smi"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              total        used        free      shared  buff/cache   available\n",
            "Mem:          12991         601        5834           1        6554       12133\n",
            "Swap:             0           0           0\n",
            "Filesystem      Size  Used Avail Use% Mounted on\n",
            "overlay          79G   43G   37G  54% /\n",
            "tmpfs            64M     0   64M   0% /dev\n",
            "tmpfs           6.4G     0  6.4G   0% /sys/fs/cgroup\n",
            "shm             5.7G     0  5.7G   0% /dev/shm\n",
            "/dev/root       2.0G  1.2G  821M  59% /sbin/docker-init\n",
            "tmpfs           6.4G   36K  6.4G   1% /var/colab\n",
            "/dev/sda1        86G   47G   40G  54% /opt/bin/.nvidia\n",
            "tmpfs           6.4G     0  6.4G   0% /proc/acpi\n",
            "tmpfs           6.4G     0  6.4G   0% /proc/scsi\n",
            "tmpfs           6.4G     0  6.4G   0% /sys/firmware\n",
            "Mon Nov 15 17:55:16 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 495.44       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   48C    P0    58W / 149W |      0MiB / 11441MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jt_PxOYPmxp_"
      },
      "source": [
        "**Questions to explore:**\n",
        "\n",
        "*   How to connect your Google Drive with Google Colab?\n",
        "*   How to import a new notebook and save it to your GDrive?\n",
        "*   How to use files which are contained in your GDrive?\n",
        "\n",
        "Some tips here: https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i74kZQufNv5d"
      },
      "source": [
        "Ok, if you are on Colab PyTorch is already installed! Let's import it and see if it can find the GPU:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hv7FUJ2Wrd_l",
        "outputId": "7374c015-0482-4714-8ed6-24cc184bbe69"
      },
      "source": [
        "import torch\n",
        "torch.cuda.is_available()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JuSqVkPnN7iT"
      },
      "source": [
        "That's great, let us import then a few libraries, which we'll be using during this tutorial!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w7AxhUWe68vT"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torchvision.datasets as datasets\n",
        "import torchvision.transforms as transforms\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rv89m9nBPXSh"
      },
      "source": [
        "# MNIST: Digits recognition with PyTorch \n",
        "\n",
        "All right, let's start then making sure we all know the basics! Let's recognize the ten handwritten digits learning from 60.000, 28x28 grayscale images.\n",
        "For simplicity let's import a loading script we have already developed inside the **Continual AI Colab** repository:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yKWbcnh474X3",
        "outputId": "ff65ad14-f1db-4b5e-c1f0-242110813fa8"
      },
      "source": [
        "!git clone https://github.com/ContinualAI/colab.git continualai/colab"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "fatal: destination path 'continualai/colab' already exists and is not an empty directory.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x3BFVukM_y8i",
        "outputId": "e3c9a181-6cbd-44e1-d681-beb090625e31"
      },
      "source": [
        "from continualai.colab.scripts import mnist\n",
        "mnist.init()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Files already downloaded!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6jIk6-G6AhWi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f6114ff9-6bd6-4c01-b4fb-7f164dc6a5de"
      },
      "source": [
        "x_train, t_train, x_test, t_test = mnist.load()\n",
        "\n",
        "print(\"x_train dim and type: \", x_train.shape, x_train.dtype)\n",
        "print(\"t_train dim and type: \", t_train.shape, t_train.dtype)\n",
        "print(\"x_test dim and type: \", x_test.shape, x_test.dtype)\n",
        "print(\"t_test dim and type: \", t_test.shape, t_test.dtype)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "x_train dim and type:  (60000, 1, 28, 28) float32\n",
            "t_train dim and type:  (60000,) uint8\n",
            "x_test dim and type:  (10000, 1, 28, 28) float32\n",
            "t_test dim and type:  (10000,) uint8\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XEWG2PmbVvb7"
      },
      "source": [
        "Let's take a look at the actual images!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RyIuYAw8AuO6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 248
        },
        "outputId": "07106299-84cf-4cef-b9ba-802d370a6fe9"
      },
      "source": [
        "f, axarr = plt.subplots(2,2)\n",
        "axarr[0,0].imshow(x_train[1, 0], cmap=\"gray\")\n",
        "axarr[0,1].imshow(x_train[2, 0], cmap=\"gray\")\n",
        "axarr[1,0].imshow(x_train[3, 0], cmap=\"gray\")\n",
        "axarr[1,1].imshow(x_train[4, 0], cmap=\"gray\")\n",
        "np.vectorize(lambda ax:ax.axis('off'))(axarr);"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAScAAADnCAYAAABcxZBBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAN30lEQVR4nO3de4hV1RfA8X0nyTRxTDM1QnuNQcVkmWUhjuRgklKikEVW4x8VSRGhEolJYU97gElWOOQDByZrMjMRkzR7qIP2gtIxy1B8YGNlZlkS3v748Vvtte3cruO956xz5vv5a23WfezyzGLvfffZJ5fP5x0AWFORdAcA4N9QnACYRHECYBLFCYBJFCcAJnUolMzlcvyUZ0Q+n88l3Ycs4dq2I+raZuQEwCSKEwCTKE4ATKI4ATCJ4gTAJIoTAJMoTgBMojgBMIniBMAkihMAkyhOAEyiOAEwieIEwKSCpxJk0cCBA1X7vvvuk/iOO+5QuUWLFkk8Z84clfvss8/K0DsA/8fICYBJFCcAJlGcAJiUK/TcuiycFjhgwADVXrNmjWp37dq1qM/55ZdfVLtHjx4n17ETxEmYpZWFa7tchg8fLnFDQ4PK1dTUSLxt27aSfB8nYQJIFYoTAJMyuZXgqquukripqUnlKisrVduf1v76668qd/ToUYnDadzgwYMlDrcV+O9DtgwdOlS1/eti6dKlcXenLAYNGiTxpk2bEusHIycAJlGcAJhEcQJgUmrXnDp37izxFVdcoXKLFy+WuE+fPkV/5vbt21V71qxZEjc2NqrcJ598IvH06dNV7qmnnir6O5Euw4YNU+2qqiqJ07rmVFGhxyjnnXeexP369VO5XC6+HS2MnACYRHECYFJqp3WvvvqqxLfeemtJPjOcHnbp0kXidevWqZw/vK+uri7J98O+8OSKDRs2JNST0gmXPu666y6J/SUS55xraWmJpU/OMXICYBTFCYBJFCcAJqVmzSk8wXLUqFESF/p5M1wrWr58uWo/99xzEu/du1flPv/8c4l//vlnlbvuuuuK+n5kS/izexbU19dH5sLtNXHK3v9pAJlAcQJgkulpnX9Q3OrVq1XOPyQuPDBv5cqVEofbDPzDspzTu7vD4W1ra6vEX375pcodO3ZMYn+K6ZzeksCDENLP3yrSq1evBHtSHuFJHb7w7y5OjJwAmERxAmASxQmASabWnPr376/aU6dOlTicFx84cEDiffv2qdzChQslPnz4sMqtWLGiYLstOnXqpNqTJ0+W+Lbbbjvpz0eybrjhBonDf+u08tfO/FMIQnv27ImjO/+KkRMAkyhOAExKfFrXsWNHif3d2s7p4XT48AH/7vDNmzerXNJD7759+yb6/Sitiy66KDL39ddfx9iT0vH/1sLtEd98843E4d9dnBg5ATCJ4gTAJIoTAJMSX3O6/PLLJfbXmEI33XSTaoenDQBJSPKhkyH/li7nnBs5cqTEEyZMULkRI0ZEfs7MmTMlPnjwYIl6d+IYOQEwieIEwKTEp3UvvPCCxOGhbf7Uzdo0zj90zD+hAO1L9+7d2/S+yy67TOLwuq+trZX4nHPOUblTTz1V4vDug/AgvCNHjkjc3Nyscn/++afEHTroMvDpp58W7HtcGDkBMIniBMAkihMAk2Jfcxo9erRq+6ddhidavvPOO7H0qS38daaw31988UXc3UEZ+Ws34b/1K6+8IvG0adOK/kz/dM1wzemvv/6S+Pfff1e5LVu2SPzaa6+pXHgbl79Ou3//fpXbvXu3xOHtXnE+OLMQRk4ATKI4ATCJ4gTApNjXnML5rb9v44cfflC5119/PZY+RfGPc3n00UcjX7dmzRrVfvjhh8vVJSRg0qRJEu/cuVPlrr322jZ95q5duyR+++23VW7r1q0Sb9y4sU2fH7r77rtVu2fPnhLv2LGjJN9RaoycAJhEcQJgUuK3r/j8LfXOHf/ggnLzp3HO6Qdu+g9bcE7/FPv888+rXPhQBWTHM888k3QX2mT48OGRuaamphh7UjxGTgBMojgBMIniBMAkU2tOSdyu4t8+E64rjR8/XuJly5ap3Lhx48rbMSAmS5cuTboL/4qREwCTKE4ATIp9Whfege23x4wZo3IPPPBAyb//wQcfVO1HHnlE4srKSpVraGiQ2H+IJ4DyY+QEwCSKEwCTKE4ATIp9zSk8SdBv9+7dW+VefPFFicNT/3788UeJBw8erHK33367xP5TLpw7/mkW/t3hq1atUrm5c+ce/x8AZIC/1tu/f3+VK9VJCCeLkRMAkyhOAEwytUP8lFNOUW3/kK9wR/ahQ4ckrqqqKvo71q9fr9pr166VeMaMGUV/DpBm/nJK+DBOK2z2CkC7R3ECYBLFCYBJsa85bdiwQbU3bdok8aBBgyLfF24z6NWrV+Rr/W0GjY2NKleOW2KANLvmmmtUe8GCBcl0JMDICYBJFCcAJsU+rfMfDOCcc2PHjpX4nnvuUTn/AQOFzJ49W7Vffvllib/99tsT7SKQeeHpIBYxcgJgEsUJgEkUJwAm5cJTAlQyl4tOIlb5fN7+IkGKtLdru66uTrX9Uz7mzZuncuHab7lFXduMnACYRHECYBLTupRgWldaXNt2MK0DkCoUJwAmUZwAmERxAmASxQmASRQnACZRnACYRHECYBLFCYBJFCcAJhW8fQUAksLICYBJFCcAJlGcAJhEcQJgEsUJgEkUJwAmUZwAmERxAmASxQmASRQnACZRnACYRHECYBLFCYBJFCcAJlGcAJhEcQJgEsUJgEkUJwAmUZwAmNShUDKXy3HAuBH5fD6XdB+yhGvbjqhrm5ETAJMoTgBMojgBMIniBMAkihMAkyhOAEyiOAEwieIEwCSKEwCTKE4ATKI4ATCJ4gTAJIoTAJMoTgBMojgBMIniBMAkihMAkwqehNneTZ8+XeLHHntM5Soq/qnrw4YNU7l169aVtV9Ae8DICYBJFCcAJjGt89TV1an2Qw89JPGxY8ci35fPc1Y+UGqMnACYRHECYBLFCYBJrDl5+vXrp9qnnXZaQj0B/ufqq69W7QkTJkhcU1Ojcpdccknk50yZMkW19+7dK/GQIUNUbvHixRI3NzcX39kSY+QEwCSKEwCTcoV+Bm8Pz5Ovra2VuLGxUeUqKyslbmlpUbnRo0dLvH//fpX7448/StlF51z08+TRNpav7fHjx0s8e/ZslTvzzDMlzuX0JfHBBx+ods+ePSW++OKLI78v/Jw33nhD4ltuueW/O3ySoq5tRk4ATKI4ATCJ4gTApHa3lSD82XT+/PkS+2tMoWeffVa1d+7cWdqOoV3p0OGfP70rr7xS5ebNmydx586dVe7DDz+UeObMmSr38ccfq3bHjh0lXrJkicqNGDEism+bN2+OzMWJkRMAkyhOAExqd9O6O++8U7XPPvvsyNf6P80uWrSoXF1CO+Tv9K6vr4983erVq1Xb32Zw6NChgt/hv7bQNG737t2qvXDhwoKfGxdGTgBMojgBMIniBMCkzN++4m/3d+74W038Ey4PHjyocjfffLPEa9euLUPvisftK6UV97Ud/uw/bdo0icO/wblz50rsP2TDuf9eZ/Jt3bpV4qqqqsjXjRs3TrWXLVtW9HeUArevAEgVihMAkzK5leDcc8+VuKmpqej3zZkzR7WTnsoh3WbMmCGxP41zzrmjR49KvGrVKpXzH6xx5MiRyM8PD0MMtwv07dtX4vDkgccff1ziuKdxxWLkBMAkihMAkyhOAEzK5JrTyJEjJa6uri742vfff1/i8NRB4ER069ZNtSdNmiRxuF3AX2caM2ZM0d9x4YUXStzQ0KByAwcOjHzfm2++qdqzZs0q+juTwsgJgEkUJwAmZWKHeDgsXrBggcSnn366yq1fv161/V3g4e5xS9ghXlrluLbPOuss1fafDRc6//zzJQ4fiDFx4kSJb7zxRpW79NJLJe7SpYvKhX/Lfnvs2LEqt3z58si+xY0d4gBSheIEwCSKEwCTUruVoK23qOzYsUO1La8zIV38W1Kcc661tVVi/wGXzjn3/fffS1xo3Tfkr2OFJxT06dNHtQ8cOCCxpTWmYjFyAmASxQmASRQnACalds3JP1bCP83yvzz99NPl6A5w3Emq/v67d999V+W6d+8u8Xfffady/hEm/p4955z76aefJG5sbFS5cM0pzKcNIycAJlGcAJiUmmndgAEDVLvQQwJ94Sl/27ZtK1mfgEKam5slDrcStNXQoUMlrqmpUblweSPcNpM2jJwAmERxAmASxQmASalZc3rvvfdU+4wzzoh87caNGyWuq6srV5eA2HXq1EnicI0pvA2GrQQAUAYUJwAmpWZa16NHD9UutCvcf9b84cOHy9YnIG7hAzizjJETAJMoTgBMojgBMMn0mtP8+fMlrqgovo6GT1gBsuL6669PuguxYeQEwCSKEwCTTE3rwpMHamtrJQ63DviHyb/00ksqx0MLkFX+wzizjpETAJMoTgBMojgBMMnUmlO3bt1Uu3fv3pGv3bNnj8RTpkwpW58ASz766COJw+01J/KgjzRg5ATAJIoTAJNMTesAFPbVV19JvH37dpULtxlccMEFEre2tpa3Y2XAyAmASRQnACZRnACYZGrNqaWlRbX90wWGDBkSd3cA05588knVrq+vV+0nnnhC4vvvv1/ltmzZUr6OlQgjJwAmUZwAmJQLn3WlkrlcdBKxyufzuaT7kCVZuLa7du2q2kuWLFFt/1SPt956S+UmTpwo8W+//VaG3hUv6tpm5ATAJIoTAJMoTgBMYs0pJVhzKq0sXtvhGpS/leDee+9VuerqaomT3lbAmhOAVKE4ATCJaV1KMK0rLa5tO5jWAUgVihMAkyhOAEwquOYEAElh5ATAJIoTAJMoTgBMojgBMIniBMAkihMAk/4GYiFm5wDOyuwAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 4 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "baEsU4PGXsgS"
      },
      "source": [
        "Good! Let's now set up a few general setting before using torch..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ztZAPQNXZ4ll"
      },
      "source": [
        "# switch to False to use CPU\n",
        "use_cuda = True\n",
        "\n",
        "use_cuda = use_cuda and torch.cuda.is_available()\n",
        "device = torch.device(\"cuda\" if use_cuda else \"cpu\");\n",
        "torch.manual_seed(1);"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Ek0mErIac6n"
      },
      "source": [
        "... and define our first conv-net! We will use 3 layers of convolutions and two fully connected layers:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ONMdybG4Be0z"
      },
      "source": [
        "class Net(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n",
        "        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n",
        "        self.conv2_drop = nn.Dropout2d()\n",
        "        self.fc1 = nn.Linear(320, 50)\n",
        "        self.fc2 = nn.Linear(50, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = F.relu(F.max_pool2d(self.conv1(x), 2))\n",
        "        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n",
        "        x = x.view(-1, 320)\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = F.dropout(x, training=self.training)\n",
        "        x = self.fc2(x)\n",
        "        return x"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p9S6a-MlYAsu"
      },
      "source": [
        "Then we can write the *train* and *test* functions. Note that for simplicity here we are not using PyTorch [Data Loaders](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html) but this is not recommended for efficiency."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HGJJfXhJB-zk"
      },
      "source": [
        "def train(model, device, x_train, t_train, optimizer, epoch):\n",
        "    model.train()\n",
        "    \n",
        "    for start in range(0, len(t_train)-1, 256):\n",
        "      end = start + 256\n",
        "      x, y = torch.from_numpy(x_train[start:end]), torch.from_numpy(t_train[start:end]).long()\n",
        "      x, y = x.to(device), y.to(device)\n",
        "      \n",
        "      optimizer.zero_grad()\n",
        "\n",
        "      output = model(x)\n",
        "      loss = F.cross_entropy(output, y)\n",
        "      loss.backward()\n",
        "      optimizer.step()\n",
        "      #print(loss.item())\n",
        "    print('Train Epoch: {} \\tLoss: {:.6f}'.format(epoch, loss.item()))\n",
        "\n",
        "def test(model, device, x_test, t_test):\n",
        "    model.eval()\n",
        "    test_loss = 0\n",
        "    correct = 0\n",
        "    for start in range(0, len(t_test)-1, 256):\n",
        "      end = start + 256\n",
        "      with torch.no_grad():\n",
        "        x, y = torch.from_numpy(x_test[start:end]), torch.from_numpy(t_test[start:end]).long()\n",
        "        x, y = x.to(device), y.to(device)\n",
        "        output = model(x)\n",
        "        test_loss += F.cross_entropy(output, y).item() # sum up batch loss\n",
        "        pred = output.max(1, keepdim=True)[1] # get the index of the max logit\n",
        "        correct += pred.eq(y.view_as(pred)).sum().item()\n",
        "\n",
        "    test_loss /= len(t_test)\n",
        "    print('Test set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
        "        test_loss, correct, len(t_test),\n",
        "        100. * correct / len(t_test)))\n",
        "    return 100. * correct / len(t_test)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IxIISdDPaqb9"
      },
      "source": [
        "Then we are ready to instantiate our model and start the training!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1cJURe0JCFh8"
      },
      "source": [
        "model = Net().to(device)\n",
        "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BlhVt8vylpUv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5ffc387f-3083-4b24-cf78-861577f6e424"
      },
      "source": [
        "for epoch in range(1, 3):\n",
        "  train(model, device, x_train, t_train, optimizer, epoch)\n",
        "  test(model, device, x_test, t_test)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train Epoch: 1 \tLoss: 0.756399\n",
            "Test set: Average loss: 0.0013, Accuracy: 9034/10000 (90%)\n",
            "\n",
            "Train Epoch: 2 \tLoss: 0.509618\n",
            "Test set: Average loss: 0.0007, Accuracy: 9427/10000 (94%)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7qwh4T5Va86-"
      },
      "source": [
        "Wow! 94% accuracy in such a short time. \n",
        "\n",
        "**Questions to explore:**\n",
        "\n",
        "*   Can you find a better parametrization to improve the final accuracy?\n",
        "*   Can you change the network architecture to improve the final accuracy?\n",
        "*   Can you achieve the same performances with a smaller architecture?\n",
        "*   What's the difference in accuracy if you change convolutions with fully connected layers?\n",
        "\n",
        "Some tips here: http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html#4d4e495354"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2dn-5gOGq08g"
      },
      "source": [
        "# Permuted MNIST\n",
        "\n",
        "But what if now we want we the same model being able to solve a new task we encounter over time like a permuted version of the same MNIST? Let's define our custom function to permute it!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Xq_4UvjgXPQ"
      },
      "source": [
        "def permute_mnist(mnist, seed):\n",
        "    \"\"\" Given the training set, permute pixels of each img the same way. \"\"\"\n",
        "\n",
        "    np.random.seed(seed)\n",
        "    print(\"starting permutation...\")\n",
        "    h = w = 28\n",
        "    perm_inds = list(range(h*w))\n",
        "    np.random.shuffle(perm_inds)\n",
        "    # print(perm_inds)\n",
        "    perm_mnist = []\n",
        "    for set in mnist:\n",
        "        num_img = set.shape[0]\n",
        "        flat_set = set.reshape(num_img, w * h)\n",
        "        perm_mnist.append(flat_set[:, perm_inds].reshape(num_img, 1, w, h))\n",
        "    print(\"done.\")\n",
        "    return perm_mnist"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4xG5LFwLgkpu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "398afb1c-4dea-421d-cab0-66b7587d49c8"
      },
      "source": [
        "x_train2, x_test2 = permute_mnist([x_train, x_test], 0)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "starting permutation...\n",
            "done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LYBa_Gedh_do",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 183
        },
        "outputId": "183cca92-ea04-414c-e77e-942fef2162bd"
      },
      "source": [
        "f, axarr = plt.subplots(1,2)\n",
        "axarr[0].imshow(x_train[1, 0], cmap=\"gray\")\n",
        "axarr[1].imshow(x_train2[2, 0], cmap=\"gray\")\n",
        "np.vectorize(lambda ax:ax.axis('off'))(axarr);"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAACmCAYAAAB5qlzZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAALAElEQVR4nO3dWUhVfRfH8X2eymYLgkabwMpuzIpCoom6ayCiwSKUuimKIIQKA7MRymgODCNSNEmxiPIigjKKyKBouqqILpqkieYoG3wuXnh532et7bN3++x19tHv5/LXOnv/PZ4WB//DjjU3NzsAABt/JXoAANCW0HQBwBBNFwAM0XQBwBBNFwAM0XQBwFD7lv4xFouxngyham5ujiXivlH4bF+7dk1kEyZMCOVekydPVvMrV66Ecj+vMjIy1Pz+/fvGI4k/t88233QBwBBNFwAM0XQBwFCspW3AUfi7F1q3tvw3Xc2oUaPU/O7du4GuW1NTo+Y5OTkiW758uciOHDmivj4rK0tkd+7c8Tk6b/76S/+O+PDhQ5Glp6eHMgY/+JsuAEQATRcADNF0AcAQTRcADNF0AcAQqxeQUMmweiEtLU1kz549CzyGyspKkeXm5ga+rh/79u0TWX5+vufXl5WViWz+/Plqbffu3UX2+/dvtVZbqVBXV6fWzp49W2QfPnxQa6urq0W2YsUKtTYoVi8AQATQdAHAEE0XAAzRdAHAEBNpSKhkmEhrDZYtW6bm2kRYshkxYoTIHjx4EPi6mZmZIrt3757n1zORBgARQNMFAEM0XQAwRNMFAEM0XQAwxOoFA2PHjhXZ6tWr1dq8vDyRVVRUqLWHDh0S2a1bt3yOLrFYvRB/2tZatwPAgyotLRVZPLbVfvv2TWSdOnUKfF1LrF4AgAig6QKAIZouABii6QKAISbS4kh7MqrjOE59fb3IUlNTA99POzO0V69ega9riYk0G+/fv1fznj17Go/k/505c0bN58yZE8r9LM8wZiINACKApgsAhmi6AGCIpgsAhmi6AGCI1Qt/aPz48SI7deqUWtu/f3+Rub3vnz59EllTU5Naq61UmDhxolqrbQ92u64lVi94U1RUJLKtW7eGcq8BAwaI7Pnz54Gvm52dLbKuXbuqtRcvXhTZ/fv31dobN26IzG1FQmNjo8j69eun1gbF6gUAiACaLgAYoukCgCGaLgAYYiLtf3Tp0kXNx4wZI7Ljx4+LLC0tTX19LCb/nu72vmsTXrt27VJrq6urPd3LcRynsLBQZDt27FBrLbWFiTTtfFvHCe+MW0QDE2kAEAE0XQAwRNMFAEM0XQAwRNMFAEPtEz2AKNGebOo4jrN48WKzMWgrJbp166bWXr58WWRTp05VazMzMwONC968fftWZGGtUli7dq2a7969O5T7WSooKBDZzp07EzASb4qLiz3X8k0XAAzRdAHAEE0XAAzRdAHAUJudSBs7dqzIZs6cqda6ba39J21iy3Ecp66uTmRukx0vXrwQ2e3bt9Xad+/eiWzatGlqrdefAZLbNl6N5fvsNpE2ffp0kfXt21etXblypcjy8/NFlpOTo75+//79Ihs6dKhaqz3hV3s6r+OE94TeoCoqKtQ8Ly9PZOvXr1dr+aYLAIZougBgiKYLAIZougBgiKYLAIZa/SHmWVlZal5fXy+y1NRUz9c9d+6cyNy2C0+ZMkVkbttyjx49KrLXr197HtevX7/U/OvXr57G5Tj6QephSYZDzBcsWCCy2trauI4nbBkZGWq+ZcsWkbmtVNBoKztGjhyp1j548EBkZWVlau2yZcs8j8GPuXPniuz06dOh3ItDzAEgAmi6AGCIpgsAhmi6AGCoVU2kDR8+XGSbNm1SaxctWiSyN2/eqLWNjY0i2759u8hOnjz5b0MMndtEmvZ7rqmpUWuXLFkS1zG1JBkm0sKi/a7atWuXgJG0Hdpnu6qqKpR7MZEGABFA0wUAQzRdADBE0wUAQzRdADCUlIeYd+zYUc21g8FnzJih1n769Elk2kHEjuM4N2/eFFnnzp1bGmJSGDRoUKKHkLTcDjb38+RfP0+Z1laa+Nmui/8Ia6WCH3zTBQBDNF0AMETTBQBDNF0AMJSU24Czs7PV/OrVq56voT0x1e1pvsnEzzbghoYGtXbSpElxHVNL2vI24FevXomsd+/eCRhJcnA7e/fChQsii8KEGduAASACaLoAYIimCwCGaLoAYCgpd6Tt3btXzWMx+Xdrt8mx1jBppnHbEeW2gwqJw6SZP/F4WOWZM2dENmfOnMDX9YNvugBgiKYLAIZougBgiKYLAIZougBgKPKrF2bNmiWyrKwstVbb6nr27Nm4jynK3FYpaO/NnTt3wh4O4kT7vdbW1qq1nLPr7tq1ayIrKChQa7X3cfTo0YHHwDddADBE0wUAQzRdADBE0wUAQ5GfSNMeAJmSkqLWaueTag/0SzZuD+LcvHmz52vU19eLbMOGDX86JBjz88DLZNKhQwc1//HjR6DrLl26VM2Li4s9X2Pnzp0i07YRO46/rcSt8zcJABFF0wUAQzRdADBE0wUAQzRdADAU+dULfnz//l1kjY2NCRjJn9NWKhQWFqq169atE9mzZ8/U2j179ojs8+fPPkeHP+G2NVtbkfDlyxe1tmvXrp7vd/v2bZHFY/tqUE1NTSJzW4kUVHl5uedaPysS3FYpaE8Xd8M3XQAwRNMFAEM0XQAwRNMFAEOtaiItmc7OdTsTWJscczsfVZsAmDdvXrCBQUhNTRXZx48fPb/ezxZePxNmbsKYNBs4cKCaP3361PM1tEmzkpIStXbVqlWer6udkTthwgS1VjtXWnuKuF8XL170XMs3XQAwRNMFAEM0XQAwRNMFAEM0XQAwFNNm8/77j7GY+z8aWbhwochOnDih1mpbYAcPHhz3MfmVn58vso0bN6q1PXr0EFlVVZVam5eXF2xgEdDc3Bx86vgP+Pls9+zZU2Tv378PPAZte3BrPaw8yurq6tR89uzZga7r9tnmNwwAhmi6AGCIpgsAhmi6AGAo8tuAtYk+t8m/vn37iuzgwYNq7bFjx0T29u1btTY7O1tkubm5Ihs1apT6+rS0NJE9efJErT1//rzI3LZKIr5KS0vVfMWKFaHcL+ikWUFBgZprT7H149evXyJr165doGvGg9vEcUVFRaDruk2Y3bt3T2SZmZmB7uU4fNMFAFM0XQAwRNMFAEM0XQAwRNMFAEOR3wa8YMECkbltA/bj5cuXInM7mHrYsGGB7tXQ0CCyS5cuqbVFRUWB7pVskmEbsKampkbN3Q6ct6Qd5u/2VOywVmZotP9zffr0Mbt/PLiNV/vZ2AYMABFA0wUAQzRdADBE0wUAQ5GfSNO20NbW1qq148aN83xd7QmgLb0X/6RtGa6urlZr16xZ4/m6bU2UJtIePXqk1qanp4c+Hvy7Dx8+qLn2/6u8vFyttTzDmIk0AIgAmi4AGKLpAoAhmi4AGKLpAoChyK9e0PTr10/NtS2NhYWFaq2f1QsHDhwQ2eHDh0XmNvsNd1FavYDW7+fPnyJr3z6cZzmwegEAIoCmCwCGaLoAYIimCwCGknIiDa0HE2nxp211raqqUmuXLFkisrC2xT5+/FhkQ4YMUWvDGoMlJtIAIAJougBgiKYLAIZougBgiKYLAIZYvYCEYvVC/GlPKrZ8SrG2esJxwluRUFlZKbLc3NxQ7uUHqxcAIAJougBgiKYLAIZougBgiIk0JFSiJtKKiorEZ3vbtm1qreUTZN1kZ2eL7Pr164GvG4WfzauSkhI1X7VqldkYmpqa1DwlJUVkTKQBQATQdAHAEE0XAAzRdAHAEE0XAAyxegEJxTZgb1i9kHxYvQAAEUDTBQBDNF0AMETTBQBDLU6kAQDii2+6AGCIpgsAhmi6AGCIpgsAhmi6AGCIpgsAhv4GzmwpIfcmmzcAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46wHcbNAchH-"
      },
      "source": [
        "Amazing. Now let's see how our pre-trained model is working on both the original and the permuted MNIST dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sxusb8s3itli",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e9af3f91-2358-4fc4-c5a5-d209d171dd71"
      },
      "source": [
        "print(\"Testing on the first task:\")\n",
        "test(model, device, x_test, t_test)\n",
        "\n",
        "print(\"Testing on the second task:\")\n",
        "test(model, device, x_test2, t_test);"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Testing on the first task:\n",
            "Test set: Average loss: 0.0007, Accuracy: 9427/10000 (94%)\n",
            "\n",
            "Testing on the second task:\n",
            "Test set: Average loss: 0.0108, Accuracy: 1028/10000 (10%)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0pHMg4G_dHFY"
      },
      "source": [
        "Mmmh... that's pretty bad, our model cannot generalize to this apparently very different new task! Well, we can just finetune our model using the new permuted training set!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J5PtR8Gqib00",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5886261f-3a1a-44b9-dc84-85e9e1e89cd3"
      },
      "source": [
        "for epoch in range(1, 3):\n",
        "  train(model, device, x_train2, t_train, optimizer, epoch)\n",
        "  test(model, device, x_test2, t_test)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train Epoch: 1 \tLoss: 1.502206\n",
            "Test set: Average loss: 0.0032, Accuracy: 7369/10000 (74%)\n",
            "\n",
            "Train Epoch: 2 \tLoss: 1.143833\n",
            "Test set: Average loss: 0.0022, Accuracy: 8253/10000 (83%)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ML7Evzb9jPAZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e419bca6-e07f-4a3c-d7fe-2cb64e950d2e"
      },
      "source": [
        "print(\"Testing on the first task:\")\n",
        "test(model, device, x_test, t_test)\n",
        "\n",
        "print(\"Testing on the second task:\")\n",
        "test(model, device, x_test2, t_test);"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Testing on the first task:\n",
            "Test set: Average loss: 0.0202, Accuracy: 2462/10000 (25%)\n",
            "\n",
            "Testing on the second task:\n",
            "Test set: Average loss: 0.0022, Accuracy: 8253/10000 (83%)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0UZ6FBuHdm7N"
      },
      "source": [
        "This is very annoying! Now we are not able to solve the original MNIST task anymore! :-( This is the phenomenon known in literature as **Catastrophic Forgetting**! In the following section we well compare three basic baselines for continual learning (and trying to not forget!)\n",
        "\n",
        "**Questions to explore:**\n",
        "\n",
        "*   When the permuted MNIST benchmark has been firstly introduced? \n",
        "*   Can simple Dropout and Regularization techniques reduce forgetting?\n",
        "*   In the permuted MNIST task, do convolutions still help increasing the accuracy?\n",
        "\n",
        "Some tips here: https://papers.nips.cc/paper/5059-compete-to-compute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9rUgLpUakTy6"
      },
      "source": [
        "## Naive Continual Learning by Finetuning\n",
        "\n",
        "Let us now try to learn continuously with just finetuning and see what we get.\n",
        "We will build a 3-tasks Permuted MNIST benchmark. Finally we will plot our accuracy results at the end of every task. Let's start by defining our 3 tasks with the function we have already introduced before:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yu0T_V24joGY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0799a0e1-7242-408e-f107-fb45554b30cb"
      },
      "source": [
        "# task 1\n",
        "task_1 = [(x_train, t_train), (x_test, t_test)]\n",
        "\n",
        "# task 2\n",
        "x_train2, x_test2 = permute_mnist([x_train, x_test], 1)\n",
        "task_2 = [(x_train2, t_train), (x_test2, t_test)]\n",
        "\n",
        "# task 3\n",
        "x_train3, x_test3 = permute_mnist([x_train, x_test], 2)\n",
        "task_3 = [(x_train3, t_train), (x_test3, t_test)]\n",
        "\n",
        "# task list\n",
        "tasks = [task_1, task_2, task_3]"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "starting permutation...\n",
            "done.\n",
            "starting permutation...\n",
            "done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gzFEA5F8pI_O"
      },
      "source": [
        "model = Net().to(device)\n",
        "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dLU6KdIbnLMN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff46575f-3f38-4a53-bd0c-db68a034bfb5"
      },
      "source": [
        "naive_accs = []\n",
        "\n",
        "for id, task in enumerate(tasks):\n",
        "  avg_acc = 0\n",
        "  print(\"Training on task: \", id)\n",
        "  \n",
        "  (x_train, t_train), _ = task\n",
        "  \n",
        "  for epoch in range(1, 2):\n",
        "    train(model, device, x_train, t_train, optimizer, epoch)\n",
        "    \n",
        "  for id_test, task in enumerate(tasks):\n",
        "    print(\"Testing on task: \", id_test)\n",
        "    _, (x_test, t_test) = task\n",
        "    acc = test(model, device, x_test, t_test)\n",
        "    avg_acc = avg_acc + acc \n",
        "  \n",
        "  naive_accs.append(avg_acc / 3)\n",
        "  print(\"Avg acc: \", avg_acc / 3)\n",
        "  "
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on task:  0\n",
            "Train Epoch: 1 \tLoss: 0.664244\n",
            "Testing on task:  0\n",
            "Test set: Average loss: 0.0013, Accuracy: 9016/10000 (90%)\n",
            "\n",
            "Testing on task:  1\n",
            "Test set: Average loss: 0.0113, Accuracy: 621/10000 (6%)\n",
            "\n",
            "Testing on task:  2\n",
            "Test set: Average loss: 0.0101, Accuracy: 1254/10000 (13%)\n",
            "\n",
            "Avg acc:  36.303333333333335\n",
            "Training on task:  1\n",
            "Train Epoch: 1 \tLoss: 1.743041\n",
            "Testing on task:  0\n",
            "Test set: Average loss: 0.0138, Accuracy: 1796/10000 (18%)\n",
            "\n",
            "Testing on task:  1\n",
            "Test set: Average loss: 0.0051, Accuracy: 6278/10000 (63%)\n",
            "\n",
            "Testing on task:  2\n",
            "Test set: Average loss: 0.0100, Accuracy: 1239/10000 (12%)\n",
            "\n",
            "Avg acc:  31.043333333333337\n",
            "Training on task:  2\n",
            "Train Epoch: 1 \tLoss: 1.569849\n",
            "Testing on task:  0\n",
            "Test set: Average loss: 0.0249, Accuracy: 1551/10000 (16%)\n",
            "\n",
            "Testing on task:  1\n",
            "Test set: Average loss: 0.0076, Accuracy: 3365/10000 (34%)\n",
            "\n",
            "Testing on task:  2\n",
            "Test set: Average loss: 0.0037, Accuracy: 7240/10000 (72%)\n",
            "\n",
            "Avg acc:  40.52\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ANJfdFD3s0oT"
      },
      "source": [
        "**Questions to explore:**\n",
        "\n",
        "*   Does the order of the tasks effect the final results? \n",
        "\n",
        "Some tips here: http://proceedings.mlr.press/v78/lomonaco17a/lomonaco17a.pdf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lCK0EYT-pJa8"
      },
      "source": [
        "## Cumulative Strategy\n",
        "\n",
        "Another simple CL idea is to carry on *all* or *part* of the previously encountered examples (of the previous tasks), shuffling them with the data of the current task. Using *all* the past data is near to the optimal performance we can desire at the end of the task sequence but at the expense of much bigger memory usage.\n",
        "\n",
        "Let's start by defining a function to shuffle our data:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FdWpT2jhfu3o"
      },
      "source": [
        "def shuffle_in_unison(dataset, seed, in_place=False):\n",
        "    \"\"\" Shuffle two (or more) list in unison. \"\"\"\n",
        "\n",
        "    np.random.seed(seed)\n",
        "    rng_state = np.random.get_state()\n",
        "    new_dataset = []\n",
        "    for x in dataset:\n",
        "        if in_place:\n",
        "            np.random.shuffle(x)\n",
        "        else:\n",
        "            new_dataset.append(np.random.permutation(x))\n",
        "        np.random.set_state(rng_state)\n",
        "\n",
        "    if not in_place:\n",
        "        return new_dataset"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94hg1UrtqFmT"
      },
      "source": [
        "Now we can reset the model and optimizer and run our training over the tasks sequence:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "62TY0Ajgbsgk"
      },
      "source": [
        "model = Net().to(device)\n",
        "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y_No-qvDbuZi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4406de47-ca62-4a6d-a669-a03c4fe6875d"
      },
      "source": [
        "cumul_accs = []\n",
        "for id, task in enumerate(tasks):\n",
        "  avg_acc = 0\n",
        "  print(\"Training on task: \", id)\n",
        "  \n",
        "  (x_train, t_train), _ = task\n",
        "  \n",
        "  # for previous task\n",
        "  for i in range(id):\n",
        "    (past_x_train, past_t_train), _ = tasks[i]\n",
        "    x_train = np.concatenate((x_train, past_x_train))\n",
        "    t_train = np.concatenate((t_train, past_t_train))\n",
        "  \n",
        "  x_train, t_train = shuffle_in_unison([x_train, t_train], 0)\n",
        "  \n",
        "  for epoch in range(1, 2):\n",
        "    train(model, device, x_train, t_train, optimizer, epoch)\n",
        "    \n",
        "  for id_test, task in enumerate(tasks):\n",
        "    print(\"Testing on task: \", id_test)\n",
        "    _, (x_test, t_test) = task\n",
        "    acc = test(model, device, x_test, t_test)\n",
        "    avg_acc = avg_acc + acc\n",
        "   \n",
        "  print(\"Avg acc: \", avg_acc / 3)\n",
        "  cumul_accs.append(avg_acc/3)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on task:  0\n",
            "Train Epoch: 1 \tLoss: 0.576622\n",
            "Testing on task:  0\n",
            "Test set: Average loss: 0.0014, Accuracy: 9051/10000 (91%)\n",
            "\n",
            "Testing on task:  1\n",
            "Test set: Average loss: 0.0113, Accuracy: 585/10000 (6%)\n",
            "\n",
            "Testing on task:  2\n",
            "Test set: Average loss: 0.0101, Accuracy: 1046/10000 (10%)\n",
            "\n",
            "Avg acc:  35.60666666666666\n",
            "Training on task:  1\n",
            "Train Epoch: 1 \tLoss: 0.860122\n",
            "Testing on task:  0\n",
            "Test set: Average loss: 0.0008, Accuracy: 9417/10000 (94%)\n",
            "\n",
            "Testing on task:  1\n",
            "Test set: Average loss: 0.0036, Accuracy: 7503/10000 (75%)\n",
            "\n",
            "Testing on task:  2\n",
            "Test set: Average loss: 0.0102, Accuracy: 1111/10000 (11%)\n",
            "\n",
            "Avg acc:  60.10333333333333\n",
            "Training on task:  2\n",
            "Train Epoch: 1 \tLoss: 0.559196\n",
            "Testing on task:  0\n",
            "Test set: Average loss: 0.0006, Accuracy: 9504/10000 (95%)\n",
            "\n",
            "Testing on task:  1\n",
            "Test set: Average loss: 0.0021, Accuracy: 8505/10000 (85%)\n",
            "\n",
            "Testing on task:  2\n",
            "Test set: Average loss: 0.0024, Accuracy: 8259/10000 (83%)\n",
            "\n",
            "Avg acc:  87.56\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jCna5k0DtN-X"
      },
      "source": [
        "**Questions to explore:**\n",
        "\n",
        "*   Can you find a way to reduce the number of examples of the previous tasks to maintain in memory? \n",
        "*   Can you find a good trade-off between memory overhead and final accuracy?\n",
        "*   Why is shuffling needed here?\n",
        "\n",
        "Some tips here: https://arxiv.org/abs/1809.05922"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RrbnV6OqW66y"
      },
      "source": [
        "## JointTraining Strategy\n",
        "\n",
        "While not a proper continual learning strategy, a commonly used baseline for continual learning is what's called \"JointTraining\" or \"offline strategy\", that is a multi-task training setting where all the data are seen at once hence simulating a static setting. This is often intended as a performance upper-bound for a continual learning problem.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gq_bYiYQbHzq"
      },
      "source": [
        "model = Net().to(device)\n",
        "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T7iwZCidbIqs",
        "outputId": "27da30f3-12cf-4153-950e-02c0416d14a4"
      },
      "source": [
        "offline_accs = []\n",
        "\n",
        "print(\"Training on all tasks together...\")\n",
        "avg_acc = 0\n",
        "(x_train, t_train), _ = tasks[0]\n",
        "  \n",
        "for i in range(1, len(tasks)):\n",
        "    (past_x_train, past_t_train), _ = tasks[i]\n",
        "    x_train = np.concatenate((x_train, past_x_train))\n",
        "    t_train = np.concatenate((t_train, past_t_train))\n",
        "  \n",
        "x_train, t_train = shuffle_in_unison([x_train, t_train], 0)\n",
        "\n",
        "for epoch in range(1, 4):\n",
        "    train(model, device, x_train, t_train, optimizer, epoch)\n",
        "    \n",
        "for id_test, task in enumerate(tasks):\n",
        "    print(\"Testing on task: \", id_test)\n",
        "    _, (x_test, t_test) = task\n",
        "    acc = test(model, device, x_test, t_test)\n",
        "    avg_acc = avg_acc + acc\n",
        "   \n",
        "print(\"Avg acc: \", avg_acc / 3)\n",
        "for i in range(len(tasks)):\n",
        "    offline_accs.append(avg_acc/3)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on all tasks together...\n",
            "Train Epoch: 1 \tLoss: 1.491086\n",
            "Train Epoch: 2 \tLoss: 0.923999\n",
            "Train Epoch: 3 \tLoss: 0.754442\n",
            "Testing on task:  0\n",
            "Test set: Average loss: 0.0007, Accuracy: 9513/10000 (95%)\n",
            "\n",
            "Testing on task:  1\n",
            "Test set: Average loss: 0.0018, Accuracy: 8709/10000 (87%)\n",
            "\n",
            "Testing on task:  2\n",
            "Test set: Average loss: 0.0018, Accuracy: 8711/10000 (87%)\n",
            "\n",
            "Avg acc:  89.77666666666666\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5B9b4cCgy0k_"
      },
      "source": [
        "**Questions to explore:**\n",
        "\n",
        "- Is the *JointTraining* strategy really an upper-bound for continual learning?\n",
        "- Can curriculum learning improve our final performace?\n",
        "\n",
        "Some tips here: https://arxiv.org/pdf/1904.03626.pdf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o3SM7U5fwTqV"
      },
      "source": [
        "## Plot Results\n",
        "\n",
        "To conclude, let's summerize our results in a nice plot! :-)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sIQEVVpDwPP5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "outputId": "571b14d3-ad29-445a-fb6e-44b2344392d8"
      },
      "source": [
        "plt.plot([1, 2, 3], naive_accs, '-o', label=\"Naive\")\n",
        "plt.plot([1, 2, 3], cumul_accs, '-o', label=\"Cumulative\")\n",
        "plt.plot([1, 2, 3], offline_accs, '-o', label=\"JointTraining\")\n",
        "plt.xlabel('Tasks Encountered', fontsize=14)\n",
        "plt.ylabel('Average Accuracy', fontsize=14)\n",
        "plt.title('CL Baselines Comparison on Permuted MNIST', fontsize=14);\n",
        "plt.xticks([1, 2, 3])\n",
        "plt.legend(prop={'size': 16});"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEbCAYAAADXk4MCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3xUxfbAvyeNNEioJiQUBRUUQTH4VARRmooNCz4FBJVnw58oVhQhWB5FsVdUlIdYeAhYUCwggs9YQDSoiAiKlFACJARIz/z+mJuw2WySTbK7Icn5fj772Xvn3jtz7t3Ze2bOzJwjxhgURVGUhktQbQugKIqi1C6qCBRFURo4qggURVEaOKoIFEVRGjiqCBRFURo4qggURVEaOKoI6iAiMlJE9pe3r/geEflLRO6sbTkU3yMi+0VkZG3LUZvUW0UgIkeIyFMiskFEckVkq4h8LCLnuZxTpT+3iLwuIsblky4iH4pIJ//chde8AxxVyzIAICJ9nGeSLiLZIvKbiDwjIu1rW7Ya0gN4vraF8AcissylTueKyO8icp+IBNe2bJ4QkfaOrEkBKi/ZKW+Jh2M3Ocd+dkkb6aR97uF8IyKXueyXegeJyJEi8oaIbHF+i20iskhETnL+W6aSz8jq3GO9VATOS+cHYCAwDugK9AMWAS/WMPvPgXjnMwCIABbUMM8aYYzJNsbsrE0ZAETkBmAJsBu4HOgMXIetZ+NrUbRqIyJhAMaYXcaYg7Utjx95DVunjwWeBh4Gqt0DEpFQH8l1uLAdOMNDg+Y64G8P5xcCZ4rIQG8LcJ7ZZ0BLYAhwDHAp8B3QDPiaQ++eeOxvluKW9o635ZXCGFPvPsBHwFYg2sOxWJftv4A7q5Dv68CHbmnnAwaIcEmbAqwDsp0ypgHhLsfbAO8Be4CDwG/AP12OJwBvA3udzyLgaJfjI4H9FewnAz8D/wQ2AFnAQqCFm+zXAL8COcDvwO1AkMvxG5z0HCAd+AQIKefZJAK5wNPlHHd97pcAa5zzNwP3A+L2u0xwnneWc84VQKzzXPYD64EBLtf0cX6H84EfHZlXASe7nNMceAvY4vw2vwDXuMm5DHgBeAzYBXzvqa5U9Gywiu8BR+5c514vcrm2vSPrpdg//kHnd+hfSf1rBDwJ7HDK/QY4w8Mz6At86+S7EuheSb7LgGfd0j4DUpztMGCq89wOAt8DAz2Uex72pZXn/A7Fz3I6tq7vAsY49/EckIF9iQ738GyS3OQxwGUu266fZVWo0x0duXKw/9HzsfVpZAXPJxn7f5oPTHJJ7+pcOx342f3/6Nzjj27ll9yHe70CTnSOd/TyffSs673X5FPvegQi0gw4B3jOGFPGbm6MyfBhWY2xL6g1xphsl0MHgGuxLeKbsS/k+12OPw9EAmcBxwO3Yf8UiEgk8AW2op4JnAakAZ87x7ylvSPbYGzP5STgERfZ/wX8G/vC7QzcAdzjyIvT7X4OmIRtJfYFFldQ3uXYF8YUTweLn7uInAz8F/unOgG4F9tru8XtktuwL5XuwFxgFvAmVsmfCCwH3hCRcLfrHnPuIwnYCHzo8tzCsT3F87HP/SngJRHp65bHMECAXsDV7vfixbMZA9zlyHECtsc4X0ROdMvqEWzruxv25fq2iES7l+fCNOxvei3291wDLBaReLfzJmOfa3ds72yOiEgF+XoiGyhu1b+GrYtXAV2wv8UHItLN7Zqp2J5fJ6wiAhiKVeb/wNaNJ7GNkt+xv9Es4BUP91ARpzjf52BbwZeAV3U6CPtbBGH/V9diX/KNvCz3VWCEkw/Y3sBc5/488SDQAfsMvGEXUARcKiIhXl7jG3yhTQ6nD7aSGGCwF+f+RdV7BAVYbb/fKedvoEsl190I/OGynwpMLOfca7GtXdcWcjD2Dz3EtcXh3gJx2U/GKpIYl7T73WQo1RJz0m4DfnW2LwEygcZePpvngUwvzpsDLHVLSwa2uP0ub7nsRzvP+mmXtPa4tBw51Cod6nZdBjCqAnneBl5x2V8GpFZUVyp7Ntje6AS3tGXAG26y3+ByPMFJO6OcPKOwLe2r3erFBuBht2fg2lrv6aQlVvAMluH0CLAvyXOwPZmp2BdZEdDW7ZqFwPNu5V7qId8Ul33Bvuzed0kLde7rMrdnU1GPoLxzKqvTA7Amm7Yux89w8hpZwfNJxvYIgp3ftj9Weex2rk/GQ4/A2Z7o1J1G7vfhXq+c/dHYhuR+4EvgIeD4cuTSHkEFVLXlU1WWY1ukJ2KVzhLgUxFpUyKAyGUi8pWIbHdm8zwBtHXJ4ylgvIikiMjDTiu5mJOBI4EsZzbDfuxLpyn2T+ktm4wxmS7724BWjnwtseapl4rLcMqZ4lLGZ8Am4E8RmSMiI5weUHl4+9w7A/9zS/sKSBCRJi5pqcUbxvbsDmJbwMXscL5bueWV4nbdGuA4ABEJFpH7RSRVRHY793wJpX8bsCaliij32Tj30LqcezzOLS3VZXtbOfdTTAfsS7MkX2NMIfZ+a5JvMdc7zyMHeB94A9vj6Y79bX91qyuDKFsfV3rI1/V3NMBOXH5HY0w+1vxZmXwV4mWd7gxsNca42vS/xSq6SnGe9yxsY+1iYJcx5qtKLpuO7YmO9rKM54A4bO/rK+Ai4EcRGe7N9dUlsN2PwLAeq3U7459B3IPGmD+Kd0RkFPZFfT3wgIicim1lTsLaJzOAC7EmCwCMMa+KyCdYm2o/4GsRmWyMSca2yH7EmpPc2VMFOfPd9g2HJgcUf9+IHYAqgzEmS0S6A72xLaBxwL9FpIcxZpuHS34HmohI63KOe4OpRP58D+dWpTFzJ9ZcMAb7MtqPNSW4v4QOVChkBc/GybPcS932S+7HGGMc6011Gmfl5ov3z+kdbJ3NBbY5L71ic4rBzppy/02y3fY9PbfKfsfitGL5il/KJQ0LLweeK63TPuI1rHJr72xXiDFmv4g8CDwkIjO9KcAYk4VVxu+LyHjs+NNDwOzqCl0Z9a5HYIzZg31wt3iyt4pIrK+LxFbeYjt0T2yr4yFjzPfGmPVAOw9ybjHGzDDGDMHaNK93Dv2AHdBKN8b84fapiiIoX2BjdmBbih08lPGHy3kFxpilxpjimVdRWPu6J+Zhu/j3ejro8tzXYp+RK2dgTUPl2VqrwqkuZUZhbdprXcr5wBgz2xjzI9asckx1Cinv2Rhj9mGfrad7/LU6ZTlswD7fknyd6Z2n1TDfYjKd339zsRJwWI19Kcd5qCtbfVCuO7ucb9cxA/exlTznu2R6q5d1ei2259nGJa9TqMJ70Pk/f8eh8Q1vmIE1I3n8b1RSnsFOJqlo7KjG1MceAdhu2P+AlSLyAFaDC3ZwdhylTQGtPQzibTHGpJeTdyMRiXO2m2IHOaOBD5y037GVbSi22z4QuNI1AxF5CvjYObcJ1iZb/Geeg225viciE7B2zzbYLuKLTkX0BROBZ0QkAzsAG4o1AyQYYyaLyPnYLvVybE/kLKAxh16qpTDGbBaR24FnRSQG21r6E2smuQrbPf4Xtqv8vYgkYwd/e2Bb6ff56L7Gi8gu7EthAval8aZz7HfgChE5AzvT5/+wZrjVVSnAi2fzKPCgiKzHmpmGYQeeu1f3powxB0TkBWCqiKRjn+3twBH4cX2DMeZ3EZkDvC4id2AbKs2w4wIbjTHzfVxetoh8A9wjIhuAGOzgtys7sb2RgSLyF5DjmEErrNPYqd+/Af9x6moE1mxbUEUxz8Xa/Pd6eU8FInIflbTonffQJOe8X7F190ysKeqtKspYNXwx0HA4frAtimewM0dysS+Gj4FzXM75i7JT0QxwSzl5vu523j5s68B9kGwytmWzHzs75iYc5e4cfwZrwspxznsbW1mLjx+BfZHudGT/E5iJM/0TL6ePuslU6hwn7UrsHzsHa6f9CmcaK7YF+wW2JZONHSy7xtNzccuzL/ZPuJtDU/SeAdq5nFM8fTSP8qeP3umWb6kpfljFYrCtcDg0YHkhVvHnOvfWw+Waps7vkeU822nYl+gyl3OW4TaV0l2myp4NpaeP5jn3erHL8fZUMiBazrN1nT6aS/nTR1tUVpZbvh7v2eV4qFOnNjr3sx1ruji5vHLLy9d5Vsluadtx+c9xaBypeFyol/uzAUZhG0mFbr9fuXXaOX4MdhA2F/sfvNC9bnm4/2Tc/k8VHcfDf81J/8bDfbjWqxZYxZSKfbfsxyqEZFymn7tc67PBYnEyVJQ6jYj0wb6cW5rye3OKonig3o0RKIqiKFVDFYGiKEoDR01DiqIoDRztESiKojRw6uT00RYtWpj27dvXthiKoih1ilWrVqUbY1q6p9dJRdC+fXtWrvS0ml1RFEUpDxHZ5CldTUOKoigNHFUEiqIoDRxVBIqiKA0cVQSKoigNnIAqAhEZIyI/i8gvInKbk9ZMRD4TkfXOd9NAyqQoitLQCZgiEJEuWO+Tp2BD850vIh2xrlmXGGOOxgZ5qbKrVm9YtHERA+YNoOusrgyYN4BFGxf5oxilgaL1S/En/q5fgZw+2hn41hhzEEBEvsR6obwI670QrH/vZdg4oz5j0cZFJH+dTE5hDgBpB9JI/joZgEFHDfJlUUoDROuX4k8CUb8C5mJCRDoD72EDaWRjW/8rsTFGY51zBNhbvO92/fU4wVvatm178qZNHqfDemTAvAGkHUgrkx4WFEbXll2rfjOK4kLqrlTyivLKpGv9UnxBefUrPiqeTy/7tEp5icgqY0ySe3rATEPGmLXYYNifAoux4RgL3c4p9vPv6foZxpgkY0xSy5ZlFsZVyPYD2z2me3q4ilJVyqtHWr8UX1BePSrvvVYdArqy2BjzKvAqgIj8G9gC7BCReGNMmojEYwOG+JS4qDiPPYL4qHheO6fSsKOKUiHl9Ti1fim+oLz6FRcV5+Hs6hHoWUOtnO+22PGBN7GRjkY4p4zAmo98ypjuYwgPDi+VFh4czpjuY3xdlNIA0fql+JNA1K9A+xp6V0SaA/nAaGNMhohMAeaKyHXAJmCIrwstHlB56oen2H5gO3FRcYzpPkYH8hSfoPVL8SeBqF91Mh5BUlKSUadziqIoVaPWB4sVRVGUapI6F57oAsmx9jt1rk+zr5NuqBVFURoMqXPhg1shP9vuZ262+wBdfWNJ1x6BoijK4cySBw8pgWLys226j1BFoCiKcrhSVGh7AJ7I3OKzYlQRKIqiHI5sXQUvn13+8ZhEnxWlikBRFOVw4uAe+PB2eLkvZKVBj1EQGlH6nNAI6DvBZ0XqYLGiKMrhQFER/PQmfDYBsvfCP26Es8ZBeAy0+YcdE8jcYnsCfSf4bKAYVBEoiqLUPtt/hkV3wOZv7Et/0HSIO+HQ8a5DfPrid0cVgaIoSm2Rsw+WTYFvX7Qt/wufhROHQlBgrfaqCBRFUQKNMfDzu/DJ/bB/B5w8AvpOhMhmtSKOKgJFUZRAkr7emoH+/BLiusI/50BiGa8PAUUVgaIoSiDIOwgrHoP/PQ2hkXDeY5B0LQQF17ZkqggURVH8zm8fwcf3QObf0PWfMOAhiG5V21KVoIpAURTFX+z9yyqA3xdDy04wchG0P6O2pSqDKgJFURRfU5BrTUArHgMJhv4Pwak3QXBobUvmEVUEiqIovmTDUlh0J+zZAMddBAMnQ0xCbUtVIaoIFEVRfMG+bfDJffDLAmh2FAx7Fzr2q22pvEIVgaIoSk0ozIdvX4Jlk6GoAM66H06/FULDK7/2MEEVgaIoSnXZ9LVdE7DzVzh6AJw7DZodWdtSVRlVBIqiKFVl/y7rHO6nNyGmDVwxBzoNApHalqxaqCJQFEXxlqJCWPWa9QSadxDOuB163wVhUbUtWY1QRaAoiuINW1dZM9C21XBkbzhvOrQ8pral8gmqCBRFUSoie6/tAax8za4GvvRV6HJpnTUDeUIVgaIoiieKiuCnt5xAMXtKB4qpZ6giUBRFcWfHL9YM9HcKJJ4CgxZAfNfalspvBFQRiMjtwCjAAGuAa4B44G2gObAKGG6MyQukXIqiKADkZsEXk2s9UEygCdjdiUgCcCuQZIzpAgQD/wSmAk8YYzoCe4HrAiWToigKcChQzLM94Jvnoftw+L9V9rueKwEIoCJwCAEiRCQEiATSgLOBec7xWcDFAZZJUZSGTPp6mH0xzLsWolrCqM/hgqdqLVpYbRAw05AxZquIPAb8DWQDn2JNQRnGmALntC2AR+9MInI9cD1A27Zt/S+woij1m7yDsGI6/O+pwy5QTKAJmCIQkabARcCRQAbwX+Acb683xswAZgAkJSUZf8ioKEoDYd3H8NHdh22gmEATyMHifsCfxphdACIyH+gJxIpIiNMrSAS2BlAmRVEaEnv/go/vhd8/PqwDxQSaQCqCv4FTRSQSaxrqC6wEvgAuw84cGgG8F0CZFEVpCBTkwtdPw/K6ESgm0ARyjOBbEZkH/AAUAKuxpp5FwNsi8rCT9mqgZFIUpQGwYSl8dBfs/gM6XwjnTIaYxNqW6rDCK0UgIhcDHxhjCmtSmDFmIjDRLXkjcEpN8lUURSlDHQ4UE2i87RHMAbJEZBbwqjHmdz/KpCiKUn1cA8UU5kOf+6DnmDoVKCbQeKsI4oCrsCuB7xSRFKwJZ64x5oC/hFMURakSm1KcQDG/OIFiptregFIhXi0oM8ZkGWNeMsacCnQFvgUmA2ki8rKInOpPIRVFUSpk/y5YcBO8dg7kZNpAMVfNVSXgJVUeLDbG/CIiTwAHgLuBK4CRIvID8C9jTKqPZVQURfFMPQ0UE2i8djEhIqEiMkREFgN/Yl1D3AgcAbQD1gLv+EVKRVEUd7b+AK/0taaguK5w0/+gX7IqgWrg7ayhZ4ArsV5DZwNjjTG/upySLSL3Att8L6KiKIoL2XthyUOwcqZdDXzJK3DCZfUqUEyg8dY0dBxwCzC/AhfR6cBZPpFKURTFHWNsoJhPH6j3gWICjVeKwBjT14tzCoAvayyRoiiKOw0sUEyg8dY09Aiw2Rjzolv6jUCCMeYBfwinKEoDJzcLlk2Bb15wAsU8AycOaxAxAgKJt09zONb9gzurgKt9J46iKApOoJj5NlBMyrNw0jAnUMzVqgT8gLdjBK2AXR7Sd2NnDSmKoviG9D/goztg4zI7G2jIbGjTo7alqtd4qwj+Bnph/QK50hsbTEZRFKVmFAeK+fppCImAcx+FHtc1yEAxgcZbRfAS8ISIhAFLnbS+2NXFU/0hmKIoDYh1H8PHd0PG39D1CusmurEaGwKFt7OGpotIC+BpIMxJzgOeMsZM85dwiqLUc/ZugsX3wrqPbKCYER/Ckb1qW6oGh9cuJowx45yYAcc5SWuNMfv9I5aiKPWakkAx0+1CsP4Pwqk3a6CYWqJKvoYcT6Pf+0kWRVEaAhu+gI/u1EAxhxFeKwIROQvrZqIth8xDABhjzvaxXIqi1Df2bYNP7odf5kPTI2Hou3C0Boo5HPBqQq6IjAQ+BhoDfbBTSZsC3YFfy71QURSlsABSnrNrAn5bZAPF3PyNKoHDCG97BHcCtxhjXhGRLGCcMWajiDwL6DiBoiie0UAxdQJvFcFRwOfOdi4Q7Ww/CywD7vWtWIqi1Gn274LPJ8KPc6BJIlzxBnQ6Xz2EHqZ4qwh2Y81CAFuBLkAq0ByI8INciqLURYoKYdXrsGQS5B3QQDF1BG8VwQpgALAGmAs8LSL9sYvKPvOTbIqi1CW2/gCLxsK21dC+FwyaDi2PrW2pFC/wVhHcAoQ725OBAqAnVik87Ae5FEWpK2igmDpPpYpAREKAfwILAYwxRahbCUXxin379rFz507y8/NrWxT/kHcAsjOg+Tlw4WXWVbQEwW+/1bZkDYaQkBDCw8Np2bIl4eHhlV/gKY/KTjDGFIjIo8CiapWgKA2Uffv2sWPHDhISEoiIiEDqUws5PxsyN0NeMIS2tQvCwiJrW6oGhzGGgoIC9u/fz99//80RRxxBTEzVI7Z569j7G+DkKufugogcKyI/unz2ichtItJMRD4TkfXOd9OalKMohws7d+4kISGByMjI+qMEigohcyvs+g3ycyCmDbQ4WpVALSEihIaG0rRpUxITE9m9e3e18vF2jOBl4DERaYsNRnPA9aAx5ofKMjDGrANOBBCRYOzsowXYqadLjDFTROReZ/8er+9AUQ5T8vPziYioJ5PqjIGcDKsEivIhsjk0bg3BVfJSo/iRiIgIcnNzq3Wtt7/im8734x6OGaCqDsP7AhuMMZtE5CLsamWAWdh1CaoIlHpBvegJ5OdA5hbIy7JxApodqdNBD0NqUte8VQRHVrsEz/wTeMvZPsIYk+Zsb6eciGcicj1wPUDbtm19LI6iKGUoKoT9O2D/TjsA3CQRolrobKB6iLfxCDb5qkAnuM2FwDgP5RgRMeXIMAOYAZCUlOTxHEVRfEROpu0FFOZBRFNokqAuousx3jqdu6SiTxXLPBf4wRizw9nfISLxTjnxwM4q5qcoiq8oyIXdG2HPRtsLaN4RmrYvUQKvv/46IkJsbCx79+4tfWlBASJCcnJylYpMTk6uHya0Ooy3pqF55aQXt8yrMkZwJYfMQgDvAyOAKc73e1XIS1EUX2CKrAkoawcIdiA4uqVVBh7IzMxk6tSpTJkypcZFjxo1inPOOafG+SjVx6segTEmyPWDjUfwD6zrid7eFiYiUUB/YL5L8hSgv4isB/o5+4qiBIqcfbDzN8hKg/DG0LKzjRdcjhIAGDBgAM888ww7duwo9xxvSUxM5NRTT61xPkr18XYdQSmMMQXGmO+B+4Dnq3DdAWNMc2NMpkvabmNMX2PM0caYfsaYPdWRSVHqOwtXb6XnlKUcee8iek5ZysLVW2uWYWEe7PkT9mwADDTrYF1Eh4RVeun48eMBePjh8j3M7Nq1ixtuuIFjjjmGyMhI2rRpw1VXXcXWraXldjcNHX/88VxySVmL83fffYeIsGDBgpK0n376iQsvvJCmTZsSERFBz549WbFiRaXyK6WpliJwIQPo4AtBFEUpn4WrtzJu/hq2ZmRjgK0Z2Yybv6Z6ysAYawbaudYOCjeOs72A8CZeZxEfH88tt9zCjBkz2LTJ81ySPXv2EB4ezuTJk1m8eDGPPvoo69evp2fPnuTk5JSb9/Dhw/noo4/KjEHMnj2bZs2aMWjQIAB++OEHTj/9dPbs2cPLL7/Mu+++S/PmzenXrx+rVq3y+l4UL8cIRKS7exIQj53vv9rXQilKfWXSB7/w67Z9Vb5u9d8Z5BUWlUrLzi/k7nmpvPXd395nZAo5rilM7BUFjZpY1xAhjaosD8A999zDSy+9xKRJk5g5c2aZ48ceeyxPPfVUyX5hYSE9e/akbdu2fPzxxwwePNhjvkOHDuX+++9n7ty53HDDDYBdnPf2229zxRVXEBZmeyx33XUXbdu2ZenSpSVpAwcOpEuXLjz00EMsXLiwWvfVEPG2R7ASG7R+pcv2+9hB4lH+EU1RlGLclUBl6WUxUJBjfQSZIhszuNlR1VYCAM2aNeOOO+7gP//5D+vWrfN4zgsvvEC3bt2Ijo4mJCSkZA1QeecDtGnThj59+jB79uyStMWLF5Oens7w4cMByM7O5ssvv+Tyyy8nKCiIgoICCgoKMMbQr18/li9fXu37aohUd0FZEbDLGFN+/05RlDJMvOD4al3Xc8pStmZkl0lPiI3gnRtOK/9CY+Dgbhs43kTamUDRcRBUVWcAnrn99tt55plnmDBhAnPmzCl17JlnnuHWW29l7NixPProozRt2pSioiJOPfXUCk1DYM1D11xzDX/++SdHHnkks2fPpmPHjpx2mr3XPXv2UFhYyEMPPcRDDz3kMY+ioiKCgmpq/W4YBHxBmaIoVeeugccybv4asvMLS9IiQoO5a2AFgV/yDloPofkHISzamoFCfev7KDo6mnHjxnHHHXdw1113lTr29ttv07dvX6ZPn16S9ueff3qV76WXXsro0aN54403uPXWW/nggw8YN+7QGtTY2FiCgoIYPXo0V199tcc8VAl4j7djBI8Am40xL7ql3wgkGGMe8IdwiqJYLj4pAYBHP1nHtoxsWsdGcNfAY0vSS1FUAPvS4GA6BIVAbDu7OthPi7ZuvvlmHn/88ZKZRMUcPHiQJk1KD0C/9tprXuXZuHFjLr74Yt544w1at25Nbm4uw4YNKzkeFRVFr169+Omnn+jevbu+9GuIt6ah4cDlHtJXYV1FqCJQFD9z8UkJnl/8xRgD2XusGaioAKJa2hlBQf71ENqoUSMmTJjA9ddfXyr9nHPOYerUqfz73//mlFNOYenSpcybV97a1LIMHz6cN998k4kTJ9KzZ0+OOuqoUscff/xxevfuzcCBA7nuuuuIj48nPT2dH374gcLCQp8sdmsoeKtGWwG7PKTvphwncYqiBJD8bNi9HjL+huAwaHGsNQX5WQkUc80113D00UeXSpswYQI33HADTzzxBIMHDyY1NZVPPvnE6zz79+9PXFwcW7duLRkkdqV79+58//33NG/enFtvvZUBAwYwZswY1qxZQ+/eXq9zVQAxpnL/bSLyO/CIMWaWW/pIYLwxpqN/xPNMUlKSWblyZSCLVJQqs3btWjp37uzfQooKIWs7HNgJEgxNWttYAeq7p0FSWZ0TkVXGmCT3dG+bCy8BTzieQ5c6aX2xgew1frGiBBoNFKP4EG9nDU0XkRbA01g/QwB5wFPGmGn+Ek5RFA9ooBjFx3jdfDDGjBORh4HjnKS1xpj9/hFLUZQyFBXB/u0aKEbxOd5OH40DQowxW7CriovTE4F8l9gCiqL4Aw0Uo/gRb2cNvYENKOPOQGC2h3RFUXxBJYFiFMUXeKsIkgBPzjtWOMcURfElpsjOBtr5mx0LaNwaWh4LjRrXtmRKPcTbMYIQwJN3qvBy0hVFqS65WZCxGQpzITzGjgV4ESNAUaqLtz2Cb4GbPKSPxmXMQFGUGlCYB3v+gt1/UNVAMYpSE7ztEdwPLBWRrhxaR3A2cBI2vKSiKNXFGDiwy4aKNMa6hYg6AtR/jhIgvF1H8I2InAbcBRTHkFsN3GyM+clfwilKvSd3v50NVJBd40AxilJdvG5yGGN+MsYMM8Yc73yGGWN+EhHtEShKVSnMhwI1JVQAACAASURBVL2brH+gogKfBIoJJCkpKQwZMoTWrVsTFhZG8+bN6d+/P7NmzaKwsLDyDALMsmXLEBGWLVtW5WuTk5NZunRpmfSRI0fSvn37mgt3GFCtvqeIJIjIeBHZCHjvRUpRGjrGwIF0Gy84ey9Et4JWnSEits4sDHvyySfp2bMne/bsYerUqXz++efMnDmTY445hptuuokPP/ywtkX0KZMmTfKoCB544AEWLFhQCxL5Hq9XFotIMHARNjRlfyAVeBH4r39EU5R6RgACxfib5cuXM3bsWG655RaefvrpUscuuugixo4dy4EDB2pJusDSoUOH2hbBZ1TaIxCRY0XkUWAb8Bjwg3NouDFmmjHGu5BDitJQKSqw00HT19mZQbHt7MKwqiqB1LnwRBdIjrXfqXP9I28FTJ06lWbNmjFtmmcXYx06dKBr164kJycjHno47uaUv/76CxHhxRdfZNy4ccTFxdG4cWOGDRvGwYMH+eOPPxg4cCDR0dF07NiRWbNmVZhfMX369KFPnz4V3sunn37KeeedR3x8PJGRkXTp0oXp06eXMm0V38MjjzyCiCAiJCcnlyk7NzeXZs2aMXbs2DLlzJ07FxFh9erVJWlffvklffv2pXHjxkRFRTFw4EB+/vnnCuX1JxUqAhFZAXwDNAWGGGOOMsaMr+gaRVFcOLjbmoEOpttAMa06Q2SzqpuBUufCB7faHgXGfn9wa0CVQWFhIV988QUDBgwgPDzcp3lPnjyZbdu2MWvWLB588EHeeecdbrzxRgYPHsygQYNYsGABXbt25ZprruGXX37xSZkbN26kb9++zJw5k0WLFjFixAiSk5O5//77S85JSUkB7Es/JSWFlJQURo0aVSavRo0aMWTIEN56660yYySzZ8+mS5cunHTSSQAsWrSIvn37Eh0dzRtvvMGbb75JVlYWvXr1YvPmzT65t6pSmWnoNOA5YIYxpsZPX0RigVeALoABrgXWAe8A7YG/sApnb03LUpSAkjoXljxoZwDFJELSdRDeHTJyIDTSrgkIi4SP74Xta6qe/5bv7QIzV/Kz4b1bYNUsz9eUR9wJcG7Vo3elp6eTnZ1Nu3btqnxtZXTo0KGktT9w4EBWrFjB7NmzmT17dkmIyqSkJN5//33mzZvH8ccfX+Myb7zxxpJtYwy9evUiLy+Pxx57jH//+98EBQVx6qmnApCQkFCyXR7Dhw/npZde4vPPP2fgwIEA7Nq1i8WLF/PII4+UnDdmzBjOPPNM3nvvvZK0s846i6OOOorp06fz5JNP1vjeqkplpqEeWGXxlYisFpHbHQd01eUpYLExphPQDVgL3AssMcYcDSxx9hWl7uCptb4k2ZqBYtpAi2OsEqgJ7kqgsvQ6xrnnlnZl1qlTJ4CSFypA06ZNadWqlc9azWlpadxwww20a9eOsLAwQkNDGT9+PBkZGezcubPK+fXs2ZMOHTowe/Yh92tvv/02RUVFDB06FID169ezYcMGhg4dSkFBQcknMjKS0047jeXLPXny8T8V9giMMauB0SJyBzZm8bXANKwCGSQiad623kUkBugNjHTyzgPyROQioI9z2ixgGXBPVW9EUWqNJQ/a1rk7EmTdRLtSjZY4YMcEMj28AGPawDWLqpdnFWnevDkRERFs2rTJ53k3bdq01H5YWFi56Tk5OTUur6ioiAsvvJBt27aRnJxMp06diIiIYOHChTzyyCPVLmPYsGE89thjHDhwgKioKGbPns3ZZ59NQoKNNV2sYK677jquu+66Mte3bdu2+jdVA7xdUJaD9TI6W0Q6YmcO3Q48LCJLjTGePJO6cyQ27vFrItING/h+DHCEMSbNOWc75cRAFpHrgeuh9h6Wongkc4vn9KIC35XRd4LtdbgqnNAImx4gQkJC6NOnD5999hm5ubk0alT+mofiMYS8vLySlzrA7t27fSpTeHg4eXl5ZdJ3795N8+bNy71uw4YNrFy5spTpCeCDDz6okTzDhw9n0qRJzJ8/n3/84x98//33pQa4i2WaPHky/fqVXYLl+qwCSZXXERhj/jDG3Au0AYZgI5V5QwjQHXjBGHMScAA3M5CxAZQ9BlE2xswwxiQZY5JatmxZVbEVxfek/QT/uZhyqqxvA8d3HQIXPG17AIj9vuBpmx5A7r33Xnbv3s3dd9/t8fiff/5JampqyTiC60yYjIwMvv76a5/K065dO3bs2MGuXbtK0jZs2MC6desqvO7gwYMAhIYecuedn5/PnDlzypwbFhZGdraHHp8HOnTowOmnn14yvhEVFcUll1xScvzYY4+lffv2/PLLLyQlJZX5dO3a1atyfE21a6oxphB4z/l4wxZgizHmW2d/HlYR7BCReGNMmojEA1U3zilKINn7Fyx9GNb8FyKawQlD4LcPyrbWw2N8W27XIQF/8bvTu3dvHn/8ccaOHcuvv/7KyJEjadu2LXv37mXJkiW88sorvPnmm5x77rnExMTwr3/9i0mTJpGbm8u0adOIjo72qTyXX345DzzwAMOGDWPs2LGkp6czefJkWrRoUeF1nTt3pl27dtx///0EBwcTGhrKE0884fHc4447jkWLFnHOOefQtGlTWrduTevWrcvNe/jw4YwePZo1a9YwePDgUvcsIjz33HNcdNFF5OXlMWTIEFq0aMGOHTv4+uuvadu2rccpqP4mYF6tjDHbgc0icqyT1Bf4FXgfGOGkjcB7xaIogeXAbjvr55kkWPsh9LoDxvwIl77subVeT+MI33bbbXz11VfExsZy5513cvbZZzNy5EjWrl3LSy+9xAUXXEBsbCwffvghQUFBDBkyhHHjxvF///d/nHXWWT6VpWPHjsybN4+tW7dy8cUXM23aNB5//HGOOeaYCq8LCwtj4cKFxMXFcfXVVzN69Gh69+7NvfeWnavy7LPPEhUVxQUXXECPHj2YMWNGhXlfccUVhISEsH37doYPH17m+Hnnncfy5cs5cOAAo0aNYuDAgdx9991s376d0047rWoPwEeItcYEqDCRE7HTR8OAjcA1WGU0F2gLbMJOH91TUT5JSUlm5cqVfpZWURzyDsI3z8P/noK8/XDSMOgzDpqU3yoEWLt2LZ07dw6QkIpSeZ0TkVXGmDLBxHxoxKwcY8yPeI5o1jeQciiKVxQWwI9vwBeTbdD4YwfZwdlWnWpbMkXxKQFVBIpSJzAG1n0En0+ybiEST4HLX4d2tdNtVxR/UxWncycANwAdgGudwd2LgU3OegNFqfv8/Q18NhE2fwPNj4Yr5kCnQXXGM6iiVAevFIGIDMAO6n6MjUxW7C2rA3aB2MX+EE5RAsaudbYHsG4RRB8B5z8JJw2HYO00K/Ufb2v5Q8BYY8zzIpLlkr4MuMPnUilKoNiXBssmw+rZEBoFZ4+HU2+utzN+FMUT3iqCLsBHHtL3AM18J46iBIicTDsLKOV5uwL4lOuh911lXUIoSgPAW0WwB0jAegd1pTt2oZii1A0KcmHlTPhyGmTvgS6X2V5AsyNrWzJFqTW8VQRvAo+KyBDsevoQETkTG6jmNX8Jpyg+o6gIfn4Xlj4EGZvgyDOh/yRofVJtS6YotY63imA88Dp2wZdgVwQLVkE8Uv5linIYsOEL+Hyi9Q0UdwIMmw8dztaZQIri4K330XxgqIhMAE7CrgZebYxZ70/hFKVGpP1kp4Ju/AJi2sLgGXDC5RAUMM8qilInqNI/whizwRgzzxgzV5WActiy9y9491/wUm9I+xEG/hv+byV0u0KVQA0ojkNcUOC9e+3imMSvv/56lctbtmwZycnJFBUVlaSNHDmyJHZwRZ9ly5ZVuTxXXGMTV1VmX5QfaLxdRzCznEMGyAH+AN4xxmzzlWCKUmUO7IYVj8H3r9igMGeMhZ5jICK2tiVrsMTHx5OSkkKHDh2qfO2yZcuYNGkS48ePJ8hR4A888ECpEJOvvPIKr776Kl999RXBwcEl6ccdd1yN5E5JSSExMbHK13Xv3p2UlJQalx9ovB0jaAn0AoqAYgfjXbDjBKuAS4AHRaSX409IUQJH3kH49gX46knrFO7EoXDWfZU6hVP8T6NGjSqN9VsVOnToUEqpLF68GIB//OMfhISU/zqrLJCOO9WVuUmTJj6930DhbT/5f9hVxYnGmN7GmN5AInZtwadAO2ARMN0vUiqKJwoLbOD2Z7rbcJHtz4CbUuCiZ+ulEli0cRED5g2g66yuDJg3gEUbAxOisjzy8/MZP3487du3JywsjPbt2zN+/Hjy8/NLzvFkGho5ciSJiYmsXr2aXr16ERkZydFHH82LL75Yck5ycjKTJk0CbPCYYpOPNxTnn5KSwumnn05ERERJIJ23336bs88+m5YtWxIdHc1JJ51UKoJYMe6moWKz2Pr16xk0aBDR0dG0a9eOBx98sJTpypNpqE+fPpxxxhl8/vnndO/encjISLp06cKCBQvKlPvWW2/RqVMnwsPDOeGEE3j//ffp06cPffr08ereq4u3imAM8KAx5mBxgrP9CHC7E394KnCi70VUFDeMgd8WwQun2/CNMW3gmsVw5Vv11jPooo2LSP46mbQDaRgMaQfSSP46uVaVwYgRI5gyZQpXX301H374ISNHjmTq1KmMGDGi0mv37dvHVVddxbBhw3jvvffo0aMHN910E1988QUAo0aNKonp+9VXX5GSkkJKSorXsmVmZvLPf/6TK6+8ko8//pirrroKgI0bN3LZZZcxZ84cFi5cyAUXXMCoUaNKKaGKGDx4MGeffTYLFy7k4osvZuLEiR4ViTsbNmxgzJgxjB07lvnz5xMfH8/ll1/OH3/8UXLOZ599xtChQ+nUqRPz58/nzjvv5LbbbuP333/3+r6ri7emoWggHljrlh7nHAPYV4X8FKV6/P0tfDbBxSncG9Dp/DozFXTqd1P5bc9vVb4udVcqeUWlo8LmFOYw4X8TmPf7vCrl1alZJ+455Z4qy+DKzz//zFtvvcXEiRNLWs4DBgwgJCSEBx54gHvvvbfCsItZWVk8//zzJYFqevfuzSeffMJbb73FWWedRWJiYomNvjKzjyf279/PG2+8wUUXXVQq/b777ivZLioqok+fPqSlpfHCCy+UGnsojzvuuINrrrkGgH79+rF06VLeeuutkrTySE9PZ/ny5Rx99NGAHUuIj49n7ty5JTJNnDiR4447jgULFpT0frp06UJSUlKlgXZqirc9ggXAqyJyuYi0dz6XA68C851zTgH8r7qUhsmu3+HtoTBzAOz90zqFu/kb6HxBnVECNcFdCVSW7m+WL18OUCrwu+v+l19+WeH1kZGRpaKVNWrUiGOOOYa///7bJ/KFhoZy/vnnl0lfv349V155JQkJCYSGhhIaGsorr7xSaYzjYgYNGlRqv0uXLl7JfPTRR5coAYBWrVrRqlWrkmsLCwtZuXIll156aSkT2Mknn8yRR/p/1bu3avZG4HHgDZdrCoCZwJ3O/lrgXz6VTlH2pcGXU+CH2RAaCWeNh9PqrlO46rbEB8wbQNqBtDLp8VHxvHZO4Bf379ljgwjGx8eXSo+Liyt1vDyaNm1aJq1Ro0bk5OT4RL6WLVuWmkUEtpfQv39/IiMjmTJlCh06dCAsLIwXXniBmTPLmxhZmmbNSrtW81Zm9+vcr01PTyc/P59WrVqVOe+II47wSraa4O2CsoPAjSJyB9b1NMAGY8wBl3N0tpDiO3L2OU7hnnOcwv2rQTuFG9N9DMlfJ5NTeOilEx4czpjuY2pFnuIX2/bt20vN4tm+fXup47WFp4HllJQUNm3axIoVKzjjjDNK0quyLsJftGjRgtDQUHbu3Fnm2I4dO2jbtq1fy6/qgrIDxphU53Og8isUpYoU5MI3L8DTJ9o1AZ0GwS3fwblTG6wSABh01CCST08mPioeQYiPiif59GQGHTWo8ov9QO/evQE7C8eVOXPmAPhklkvxdM/s7Owa5wVw8KCd6xIaGlqStnfvXt577z2f5F8TgoODSUpK4t1338U1jvyqVav4888//V5+VSKUnQVciQ0yH+Z6zBhzto/lUhoaRUXwy3w7DVSdwnlk0FGDau3F74qI0KVLF6688kqSk5MpKCjg9NNPJyUlhYceeogrr7ySE044ocblFC/Kmj59Oueee27Jy7K6nH766TRp0oTRo0czadIkDhw4wMMPP0yLFi3IzMyssbw1ZdKkSQwYMIDBgwdz/fXXk56eTnJyMnFxcSUL6vyFV7mLyEjsOoLGQB9gF9AU64b6Vz/JpjQUNnwBL/eBd6+DRk1g2Ltw9XuqBA4zsrOzCQ4OLrG9v/7669xzzz3MnDmT8847j1dffZV77rnHq+mU3nD++edz88038/zzz3PaaafRo0ePGuXXsmVLFixYQGFhIZdddhnjxo1j1KhRZQa8a4v+/fszZ84c1q5dy+DBg5k6dSrTp08nLi6OmJgYv5Ytrt2Qck8S+Rl40hjzihOhrJsxZqOIPAvsN8bc61cp3UhKSjIrV64MZJGKP0hLtV5BNyy1TuHOHl+vnMKtXbuWzp0717YYPuOSSy4hNTW11Nx3xb9s2bKFjh07cv/99/PAAw9Uen5ldU5EVhljynSrvDUNHQV87mzncmjtwLPYcJUBVQRKHWfvJlj6MKyZCxFNrVO4pOsgNLy2JVM8sHLlSlasWMGiRYsYO3ZsbYtTb8nOzmbs2LH069ePFi1asHHjRqZNm0ZkZCSjRo3ya9neKoLdWLMQwFasn6FUoDmHAtkrSsUc3APLH4PvX3acwt0OPW9Tp3CHOUOGDKGoqIgxY8aUuH1QfE9wcDDbt2/nlltuYffu3URFRdGrVy/++9//lpmm62u8VQQrgAHAGmAu8LSI9Af6Ap/5STalvuDJKVyfcRCTUNuSKV6wcePG2hahQRAWFubR/1Ag8FYR3AIU99snYxeT9cQqhYe9LUxE/gKygEKgwBiTJCLNgHeA9tiYyEOMMXu9zVM5jCksgB/nwLLJkJUGx5wL/SZCq/pjN1eU+kClikBEQoB/AgsBjDFFWAdz1eUsY0y6y/69wBJjzBQRudfZr5kjFKV2MQbWfQyfJ0P6OkjsAZfNhHan17ZkiqJ4oFJFYIwpEJFHsW6m/cFF2CmpALOwg8+qCOoqf39rZwL9nQLNO8KQ2Q3GH5AnjDFeu09WlJrgzQzQ8vDWNPQNcDI2eH1NMMCnImKAl4wxM4AjjDHFTlS2Ax4da4jI9cD1gN+XWyvVYNfvsGQS/PYhRB8B5z8BJw2H4NDKr62nhIaGkp2dTWRkZG2LojQAsrOzqxR8xxVvFcHLwGMi0hYbkayUewljzA9e5nOGMWariLQCPhORUv54jTHGURJlcJTGDLDrCLwsT/E3WdvtGMAPsyE0As66H04bXWedwvmSVq1asXXrVhISEoiIiNCegeJzjDEUFBSQlZVFenp6tR3UeasI3nS+H/ckCxDsIb3sicZsdb53isgCrOvqHSISb4xJE5F4oKzXJeXwo9gp3DfPQ2E+9BhlncJFt6xtyQ4bmjRpAsC2bdtKRe1SFF8SEhJCeHg4bdu2JTy8emtxvFUENXaILSJRQJAxJsvZHgA8CLwPjACmON+17wFKKZ+CPFg5E5ZPg4O7oculdkVws6NqW7LDkiZNmpQoBEU5XPHWDXVNxwbA2v4XON3jEOBNY8xiEfkemCsi12HHIIb4oCzF15RxCtcb+k2ChO61LZmiKDWkKt5HzwVGY91NDDTGbBaRUcCfxpgllV1vjNkIdPOQvhu7ME05XNm4DD6bCGk/whEnWKdwHfo22JlAilLf8Nb76FDs4rH1WDNR8VSQYOBu/4im1DppqTB7MPznImsGGvwS3LAcOvZTJaAo9QhvewR3A/8yxrzt9AKK+QZr51fqE3s3wRePQOpc6wdowCN2MFidwilKvcRbRXA0kOIhfT+gI2H1BXencD3HWMdw6hROUeo13iqCbcAxlF1Q1hvY4FOJlMCTdxC+fdFxCpcFJ14Ffe5Tp3CK0kDwVhHMwHocLTYLtRGRXsA0INkfgikBoLAAfnoTvpgMWdvUKZyiNFC8nT46TURisC6nw4EvsAFqHjPGPOdH+RR/UOwUbskk2PWbdQp36SvQvmdtS6YoSi3g9fRRY8z9IvIIcBx2ttGvxpj9fpNM8Q+bv4PPJqhTOEVRSvBKEYjIbdgFYDsBDRZcF0lfb91C//YhRLWCQY9D96sbtFM4RVEs3vYIxgLTRGQpMBtYYIw56D+xFJ+RtR2WTYEf/nPIKdypN0Oj6MqvVRSlQeCtImiHjRlwFfAM8KKIvAe8AXzqBKtRDidy9sHXT0PKc1CYp07hFEUpF28Hiw12gPgLERkNnI9VCvOBDKC13yRUqoa7U7jjL4G+D6hTOEVRysXrweJijDF5IpKCdTVxPHCsz6VSqk6xU7ilD8Hev6B9L+g/CRJOrm3JFEU5zKmK07nGwGXAUOBM4A9snII3/COa4jWlnMJ1gaHvQkd1Cqcoind4O2toHnAesA94BxhnjPnen4IpXrB9jVUAG5ZATBvrFO6EyyHIqzhBiqIogPc9glzgUuzAcKHrARHpZ4z53OeSKeXj6hQuPAYGPAw9/qVO4RRFqRbeDhYPdd0XkQTgGuBa7IwibYIGgoN7YMV0+G6Gi1O42yCiaW1LpihKHaYqYwTBwEXAKKA/kAq8CPzXP6IpJeRnwzcvuDmFGwcxibUtmaIo9YBKFYGIHIt9+V8NHMAOEPcHhhtjfvWveA2cMk7hzoG+E+GI42pbMkVR6hEVKgIRWQF0Ad4FhhhjvnTS7wmAbA0XY+D3xdYlxK7fICFJncIpiuI3KusRnAY8B8wwxvwSAHmUzd/ZmUB/fw3NOsCQ/0DnC3UqqKIofqMyRdADaxb6SkT+Av4DvOVvoRok6eutW+i1H6hTOEVRAkqFisAYsxoYLSJ3AJdjZwlNw7qhHiQiacaYvf4Xsx7j7hSuz31w2mh1CqcoSsDwdvpoDtbr6GwR6YjtJdwOPCwiS40x5/pRxvpJGadw10Hvu9UpnKIoAac6vob+AO4Vkfuxzueu9blU9ZmCPFj1Gnw5DQ6mw/GD4ewHoHmH2pZMUZTDlIWrt/LoJ+vYlpFN69gI7hp4LBef5LuY4lVWBMU4K4zfcz5KZahTOEVRqsHC1VsZN38N2fnWqcPWjGzGzV8D4DNlEOSTXKqAiASLyGoR+dDZP1JEvhWRP0TkHREJC7RMfmfjl/DyWfDudRAaBUPnwYgPVAkoilIhxhgmf7y2RAkUk51fyKOfrPNZOdXuEdSAMcBaoImzPxV4whjztoi8CFwHvFALcvked6dwF78IXYeoUzhFUTyyMyuH1M2Z/LQlg5+2ZJK6JYOMg/kez92Wke2zcgOqCEQkERgEPAKMFREBzsYGuQGYBSRT1xVBxt+w9BFIfUedwimK4pGsnHzWbM0kdUsmP23O4KfNGWzLzAEgOEg45ojGnHN8HIt/3k5Gdlll0Do2wmeyBLpH8CRwN9DY2W8OZBhjCpz9LYBHo5eIXA9cD9C2bVs/i1lNXJ3CIdDzVjjjdnUKpygNnLyCIn7bvo+fNmfw42bb0v9j136MscfbNovk5PbNuDYxhm5tYjm+dRMiw+zr+dSjmpcaIwCICA3mroG+iwkWMEUgIucDO40xq0SkT1WvN8bMAGYAJCUlGR+LVzPys+HbF2HFE5C7D04cCmepUzhFaYgUFRk2ph+wrXzHxLN22z7yCm1o9xbRYXRLjOX8rq3p1iaGromxNIsqf2i0eED4sJw1VA16AheKyHlAOHaM4CkgVkRCnF5BIrA1gDLVjKJC+PFN+OLf6hROURogxhi278txXvrWxLNmSyZZudbIERUWTJeEGK7p2Z5ubWLpmhhDQmwEUkWXMReflODTF787AVMExphxwDgAp0dwpzFmqIj8FxsC821gBHVhOmoZp3Anw6UvQ/szalsyRVH8SObBfFK3ZpQy8ezMygUgJEjoHN+Ei05qTdfEWE5sE0uHltEEBx3+fsJqY9aQO/cAb4vIw8Bq4NValqdiNn8Pn0045BTu8llw3EXqFE5R6hk5+YX8ss3a9VMdE8+f6QdKjh/VMoqeHVvQzbHrd45vQnho3ZwRWCuKwBizDFjmbG8ETqkNOapEGadw06H7CHUKpyj1gMIiw/qdWaVMPOu2Z1FQZIcj45qE0zUxhstOTqRbYiwnJMYQE1F//vuHQ4/g8CZrO3w5FVbNUqdwilIPMMawZW+2HcjdnMFPmzP5eVsmB/PsrJzG4SF0S4zlhjOPomtiLN0SY4mLqd9Tv1URlEduFvzvaUh51jqFS7oWzrwbolvVtmSKolSB3ftzSd2SyY8uJp49B/IACAsJ4vjWTRiS1IZubWLolhhL++ZRBNUBu74vUUXgTkEerHrd9gLUKZyi1CkO5Bbws7NI60enxb9lr12BKwJHt4qmb6dWdGtjW/rHxjUmLCTgnnYOOxqMIqjUe19REfy6AJY8BHv/VKdwinKYk19YxLrtWSUmntQtmfy+IwvHrE9CbAQntoll+Knt6NYmli4JMUQ3ajCvvCrRIJ5Kpd77Nn4Jn0+Ebauh1fHWKVzHfjoTSFEOE4wx/LX74KFFWpsz+GXbPnIL7CKtppGhdE2MZcDxcZzoLNJqEd2olqWuOzQIRfDoJ+s8eu/776KPuWDNRwRvXAJNEtUpnKIcJuzcl1Mye+enLba1n+n42wkPDeKEhBiGn9qOrm1iOTExljbNqr5ISzlEg1AE2zKyuTDoK+4OmUtrSWenacomWtIjfz37NkTyevBwVoReQss1MbTevI7WseG0jo1wPuG0iGrU4AaPFCVQ7MvJ52cXm37qlkzSXJyvHXtEY847IY5uibF0axPL0a2iCQlWu74vaRCKYET0d9yd/wqRYmcKxMlejjB7+VK68/upj7HjYChRGTn8sWs/y9fvKplGVkxYcBBxMeElCiLBURLxMeElQ7YOKAAADuZJREFU21Fqe1SUSsktKGRtWlYpE8+GXYcWabVvHkmP9s2cwdwYjm8dQ0SY9tD9TYN4e90d+g6RBXml0kTglIg0+pxTejDYGMO+7AK2ZmSzLSObtMxstmbksM3Z/3bjHrbvy6GwqLTfu5iI0FKKobg3Ubx9RONG2opRGhRFRYYNu/aXMvGsTdtHfqH977SIbsSJbWK4+MQEuraJpWtCDE0rcL6m+I8GoQgis7d7nS4ixESGEhMZynGtm3i4CgoKi9iZlcu2jGy2ZmSTlnlIUWzNyGHlpr0l9sxigsSuTox3URIJsRG0jokg3tmOiQhVO6dSJzHGkJZpna/9uCWD1M2ZrNmayX7H+Vp0oxBOSIjh2jOO5ETHxBMfE671/TChQSgCYhIhc7Pn9GoQEhxU0tJPKuecA7kFpXoTaRmHtlO3ZPDJzzklbmmLiQwLLmNyah0bQesY27OIiwmvs75MlPpFxsE8G0HLaen/uDmT9P3W+VposHW+NvikhBITz1F1xPlaQ6VhKIK+E+CDW23cgGJCI2y6n4hqFELHVo3p2Kqxx+NFRYbdB/JcehKlexZr07JK/liutIhuREJsOPExbj2LWNuz0IFtxddk5xXyy7bMUiaeTbsPlhzv0DKK3se0KBnM7RzfmEYh2mCpSzQMRdB1iP1e8iBkbrE9gb4TDqXXAkFBQsvGjWjZuBHd2sR6PCe3oJDtmTnOeEVOqTGLiga242PDS5mcinsWxQpEB7aV8igoLGL9zv0ug7mZrNuRVTImFh9jna9d0aMNJybG0iUxhibh9cf5WkNFjDm8gn15Q1JSklm5cmVti1HrGGPIzM4vURLbMp2ehcvg9vZ9ObiNaxMTEVpKMbj3LFrpwHaDwBjD5j3Zjk3fvvh/3rqvZM1Nk/CQElcMxSaeVk3qt/O1+o6IrDLGlLFoa9OwDiMixEaGERsZVuHA9o6sXGeMomzP4vu/yh/Ybu1icioe2C6ePtskIkQH+uoY6ftzSXXs+cU+9vcetL99WEgQXVo3sS39NvbF3755pP7GDQRVBPWckOAgEpyXd3kD2/tzC0jLyGZbqdlPtmfx05YMFlcwsF1ezyIuJlztxLXIgdwC1mzNLFmg9ePmDLZm2DGyIIFjjmj8/+3dfXBU1RnH8e8vIQEMMaGAGBIEFQcVkaAjolY6tb626FC003asSuu0/aOd6ljtH61TX2pLHbXajlVLlYov1VrU0WLtqFW0KmAtAkXwFXWERAyVJLwFEJ7+cc6Gm80mIbjZTbLPZ+YOu3fvPffZ7OU+e865ew6nHTmyzeBrJV4LLFieCBxDBg7gsJHlHDay447tDVu2U9fYkrFmsaquudOO7daaRdrdUMPKSr1jOwt2fBoGX0s28bzz8ebWJsGaoYOpPaiSWSeO5eiaCo6qrvB+IteGnw2uS0VF4oDyQRxQPojaDjq2W3aGju26DDWLt9ZvYuGbDe3Ge0p2bLfWLLxju1O7dxvv/W9LGFf/w/BNf1V9Mzvi4GufKytlUk0FZx1VRW2cLH2YD77muuD/y1xWDCopZuzwMsYOL8v4eqpjO1WbqG9qW7N4+d0NrO+iYztTzaK/d2yvb25h2Yd7xuBZvraRTS3hR1qDS4qZWF3BRSeMaW3iqRnqg6+57vNE4HIi2bE9YVRFxm1SHdup2kRd4u6ntRu38cp7n9AcL4IpxUViZPnAxLAebfss+lLHdtO2nfw3XuxTt2+ubw5NbsVF4vADyzl70qjWydLHjfDB11x2eCJwvUayY7sjqY7t9jWLbSz7sJEnV9a3jmWTUlZa3KbJKdUUVZXHju2WnbtYVd8c2/TDxX9NYvC1g4eXMfWQYa23bk4Ytb//qtz1GE8Erk/pTsd2u5pF0zZW1TWxYfOOdvuNKB/YOpRHpprF8CGlndYqOpsBb1ccfC3ZxLO6vplPYzvYiPKBTKqpZGYckuHo6koq9vMfabnc8R+UuYKT7NjOVLOoa2xp37E9oIhRFckmp0GxVjGYN+qbuPmZt2nZuecW25JicfK44WzZsYuV65rYEn8BPmTgAI6uCTNo1Y4OTTwH7u+Dr7nc8B+UORftTcd249ad1DW17adI3Q3VUcd20s5dxrNvNjBpdCXnHlsTm3gqOGT4EL9l1vU6OUsEkgYBLwAD43Hnm9lVkg4GHgSGAf8BLjCz9nV353JEEkPLShla1nHH9s5du1nf3EJ9Uwtfu2NR5nKAx35wUg9G6lx25PKWg+3AKWY2CagFzpQ0FbgeuNnMxgEbgYtzGJNz+6SkuIiaoWE2rY46t0d10untXG+Ss0Rgweb4tCQuBpwCzI/r5wEzchWTc9lwxRnjGZx2R8/gkmKuOGN8niJyrntyehOypGJJy4CPgaeBd4FGM0vdHL4WqO5g3+9JelXSqw0NDbkJ2Lm9MGNyNbNnTqS6cjACqisHM3vmxNa7hpzr7XLaWWxmu4BaSZXAo8Dh3dh3DjAHwl1DPROhc/tmxuRqv/C7PisvP0s0s0bgOeAEoFJSKiHVAOvyEZNzzhWqnCUCSSNiTQBJg4HTgNWEhHBe3Owi4LFcxeSccy63TUNVwDxJxYQE9JCZLZC0CnhQ0nXAa8BdOYzJOecKXs4SgZmtACZnWL8GmJKrOJxzzrXlQxc651yB65NjDUlqAD7Yx92HAxuyGI5zSX5+uZ70Wc+vMWY2In1ln0wEn4WkVzMNuuRcNvj55XpST51f3jTknHMFzhOBc84VuEJMBHPyHYDr1/z8cj2pR86vgusjcM4511Yh1gicc84leCJwzrkCVzCJQNJcSR9LWpnvWFz/I2m0pOckrZL0uqRL8h2T6z8kDZL0iqTl8fy6JqvlF0ofgaRpwGbgHjM7Kt/xuP5FUhVQZWZLJZUTpl2dYWar8hya6wckCSgzs82SSoAXgUvMbHE2yi+YGoGZvQB8ku84XP9kZvVmtjQ+3kQYWdcnKHBZ0ckMj1lRMInAuVyRNJYwwOKS/Ebi+pP0GR7NLGvnlycC57JI0hDgYeBSM2vOdzyu/zCzXWZWS5jAa4qkrDVxeyJwLkti2+3DwP1m9ki+43H9U2KGxzOzVaYnAueyIHbm3QWsNrPf5Dse1790MMPjG9kqv2ASgaQHgEXAeElrJV2c75hcv3IScAFwiqRlcflyvoNy/UYV8JykFcC/CX0EC7JVeMHcPuqccy6zgqkROOecy8wTgXPOFThPBM45V+A8ETjnXIHzROCccwXOE4HrEyRd7SPH9m6SbpW0MN9xuO7zROCyRpJ1sdyd7xhTJM3qJM5B+Y6vuyS9L+nyfMfh+qYB+Q7A9StVicfTgT+mrduW23C6tBU4NH2lmbXkIZZeQVKpme3Idxwut7xG4LLGzD5KLUBjch1QBtwj6SNJWyQtlTQ9ub+kmZJWSNom6RNJz0samelYkg6S9IakeZIGSKqQdG+cfKhF0hpJl3Yd8p6YE7GmjrFQ0m2SfiVpQyz7RklFiW1K4+sfSNoej/ujxOvTJC2JMa2XdLOk0rRj3Jr23u6WtCBtmw7jiM0xY4AbUrWaxL4nxr/jVknrJN0uaf+0sm+P5TUAL8X1R0p6QtKmeLwHJB2Y2K847rMxLrcAxV38vV0v5YnA5coQ4EnCGCmTCIOzPSLpcIB4kXkQmAccAUwD7s1UkKQjCBesvwOzzOxT4DpgIqEmMh74DrAuC3GfD3wKnAj8ELgU+Hri9XnAhcBlMe6LiUlQUnV8z68RhqW+GPgmMDvLccwE1gLXEmpgVfH4E4GngMcJf/OZQC0wN63sbwECTgYuVJhk5wVgJTAFOJXw+T2WSII/Br4LfB84gZAEzt+H9+V6AzPzxZesL8B54fTqdJvFwJXx8TGEiTbGdLDt1YQL0/HABuBnaa8/DsztRnyz4vE2py0vJ7ZZCCxK2+9p4M74+LBYxpkdHOOXwNtAUdpxtwP7JY5xa9p+dwML9jaO+Px94PK0be4B7kpbVxtjPiBR9oq0ba4F/pm2bmjcb0p8Xpf8DAhfKt8CFub73POl+4v3EbickFQGXEX4xl5FmGFpELAibrIceAZYKemp+Hi+mTUkiqmO6681sxvSDnE7MF/SsYSL5N/M7PkuwtpKuDAmbU97viLteR1wQHw8GdhNGBI4kyOAxWa2O7HuRaAUGJeh7M50FkdHjgXGSUrWYBT/PZQwwQmEaTXT95smaTPtHSrpTcJnuCi10sx2S1oCjO4iJtcLeSJwuXIjYfz0ywnfkrcSvrGWQph0Q9LpwFTgdEIzymxJXzCz5bGMDYRvvt+QdKeZbUwVbmZPShoDnAV8CXhC0l/N7NudxGRm9k4Xce9M34fsNKmm2vF3s+finFKSpTiKgDuBmzO8lmw225JhvycIn1W69XtxXNfH+AfqcuXzwD1m9rCZrSC0abe5Y8eCRWZ2DXAc4Vtv8tvsduAcYCPwtOL47In9N5jZvWY2i5BILpI0sMfeESwj/B/6YgevrwamJjuXCX+HHcC78XkDbe+sgtCe3107aN9ZuxSYYGbvZFg6u4NrKTAB+CDDfpvMrAmoJyRtoHU+hin7ELfrBTwRuFx5C/iqpGNiJ+Z9hKYhACRNlXSlpOMkHUS44I8GViULiRews4EmEslA0rWSZkg6LHYmzwTWmFl6U0+SJB2YYdmru1/M7C3gIeBOSedKOljSyZIuiJvcBowCbpN0hKSvAL8m9Alsjds8C5wl6RxJ4yX9hn1rXnkfOFlStaThcd31hCkN75A0WdI4SdMl/aGLsn4PVAB/kXS8pEMknSppjqTyuM1vgZ9IOk/SeOAW2ic010d4InC5chmhTfpfhDtpFsfHKU2EyV0WEJqObgJ+YWb3pRcUk8F0oJk9yWA7oXN2OeGOonJCwujMfoRvtunLwd14XxcCfwZ+R5gx6m7CRRQzW0doqppMqD3MBR4AfprYf25ieQnYBDzajeOn/JyQQN4l1DKINa9pwFjgecLfZjaheadDZlZH+Cx2A/8AXickh+3s6UO5CfgToelpCeFacv8+xO16AZ+YxjnnCpzXCJxzrsB5InDOuQLnicA55wqcJwLnnCtwngicc67AeSJwzrkC54nAOecKnCcC55wrcP8Hfwmcs+UhR+sAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9x1D3O7iunxS"
      },
      "source": [
        "**Questions to explore:**\n",
        "\n",
        "*   What's the difference in terms of memory utilization among the three methods? \n",
        "*   Can you plot a similar graph highlighting the memory increase over time?\n",
        "\n",
        "Some tips here: https://stackoverflow.com/questions/449560/how-do-i-determine-the-size-of-an-object-in-python/30316760"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5_wUBAosdjUe"
      },
      "source": [
        "# Split MNIST\n",
        "\n",
        "Split MNIST is just a split in different batches of the original MNIST data. You can do this definining a simple function such as the following:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FiL1p163d5CJ"
      },
      "source": [
        "def split_mnist(train_x, train_y, test_x, test_y, n_splits=5):\n",
        "    \"\"\" Given the training set, split the tensors by the class label. \"\"\"\n",
        "    n_classes = 10\n",
        "    if n_classes % n_splits != 0:\n",
        "        print(\"n_classes should be a multiple of the number of splits!\")\n",
        "        raise NotImplemented\n",
        "    class_for_split = n_classes // n_splits\n",
        "    mnist_train_test = [[],[]]  # train and test\n",
        "    for id, data_set in enumerate([(train_x, train_y), (test_x, test_y)]):\n",
        "        for i in range(n_splits):\n",
        "            start = i * class_for_split\n",
        "            end = (i + 1) * class_for_split\n",
        "            split_idxs = np.where(np.logical_and(data_set[1] >= start, data_set[1] < end))[0]\n",
        "            mnist_train_test[id].append((data_set[0][split_idxs], data_set[1][split_idxs]))\n",
        "    return mnist_train_test"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "89m-P1b_iBF9"
      },
      "source": [
        "train_x, train_y, test_x, test_y = mnist.load()\n",
        "splitmnist = split_mnist(train_x, train_y, test_x, test_y, n_splits=5)"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9i8D6_dioCYf",
        "outputId": "3c7e6c0b-a90b-4ed3-e6ac-31597d0c7349"
      },
      "source": [
        "for i in range(5):\n",
        "    train_split_x, train_split_y = splitmnist[0][i]\n",
        "    test_split_x, tests_split_y = splitmnist[1][i]\n",
        "    print(\"train & test splits for task {}: \".format(i))\n",
        "    print(train_split_x.shape)\n",
        "    print(train_split_y.shape)\n",
        "    print(min(train_split_y), max(train_split_y))"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train & test splits for task 0: \n",
            "(12665, 1, 28, 28)\n",
            "(12665,)\n",
            "0 1\n",
            "train & test splits for task 1: \n",
            "(12089, 1, 28, 28)\n",
            "(12089,)\n",
            "2 3\n",
            "train & test splits for task 2: \n",
            "(11263, 1, 28, 28)\n",
            "(11263,)\n",
            "4 5\n",
            "train & test splits for task 3: \n",
            "(12183, 1, 28, 28)\n",
            "(12183,)\n",
            "6 7\n",
            "train & test splits for task 4: \n",
            "(11800, 1, 28, 28)\n",
            "(11800,)\n",
            "8 9\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1UBJxBqExQJy"
      },
      "source": [
        "**Exercises / Questions to explore**:\n",
        "\n",
        "- Plot a few sample data belonging to each experience.\n",
        "- Is **SplitMNIST** harder than **PermutedMNIST**?\n",
        "- Is it more realistic? Why?\n",
        "- Compute the results of the introduced baselines. What can you deduce from the plots?\n",
        "\n",
        "Some tips here: https://arxiv.org/pdf/1904.07734.pdf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OS9gHzZM7HQr"
      },
      "source": [
        "**Copyright (c) 2021. Continual AI. All rights reserved.**\n",
        "\n",
        "See the accompanying LICENSE file in the GitHub repository for terms. \n",
        "\n",
        "*Date: 15-11-2021                                                             \n",
        "Author: Vincenzo Lomonaco                                                    \n",
        "E-mail: contact@continualai.org                                           \n",
        "Website: continualai.org*                                               "
      ]
    }
  ]
}