lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Perceptron\n",
    "\n",
    "> This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners). Visit the repository for complete set of learning materials.\n",
    "\n",
    "As we have discussed, perceptron allows you to solve **binary classification problem**, i.e. to classify input examples into two classes - we can call them **positive** and **negative**.\n",
    "\n",
    "First, let's import some required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pylab\n",
    "from matplotlib import gridspec\n",
    "from sklearn.datasets import make_classification\n",
    "import numpy as np\n",
    "from ipywidgets import interact, interactive, fixed\n",
    "import ipywidgets as widgets\n",
    "import pickle\n",
    "import os\n",
    "import gzip\n",
    "\n",
    "# pick the seed for reproducability - change it to explore the effects of random variations\n",
    "np.random.seed(1)\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Toy Problem\n",
    "\n",
    "To begin with, let's start with a toy problem, where we have two input features. For example, in medicine we may want to classify tumours into benign and malignant, depending on its size and age.\n",
    "\n",
    "We will generate a random classification dataset using `make_classification` function from SciKit Learn library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features:\n",
      " [[-1.7441838  -1.3952037 ]\n",
      " [ 2.5921783  -0.08124504]\n",
      " [ 0.9218062   0.91789985]\n",
      " [-0.8437018  -0.18738253]]\n",
      "Labels:\n",
      " [-1 -1  1 -1]\n"
     ]
    }
   ],
   "source": [
    "n = 50\n",
    "X, Y = make_classification(n_samples = n, n_features=2,\n",
    "                           n_redundant=0, n_informative=2, flip_y=0)\n",
    "Y = Y*2-1 # convert initial 0/1 values into -1/1\n",
    "X = X.astype(np.float32); Y = Y.astype(np.int32) # features - float, label - int\n",
    "\n",
    "# Split the dataset into training and test\n",
    "train_x, test_x = np.split(X, [ n*8//10])\n",
    "train_labels, test_labels = np.split(Y, [n*8//10])\n",
    "print(\"Features:\\n\",train_x[0:4])\n",
    "print(\"Labels:\\n\",train_labels[0:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also plot the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-4-ade59d2ee664>:11: UserWarning: Matplotlib is currently using module://ipykernel.pylab.backend_inline, which is a non-GUI backend, so cannot show the figure.\n",
      "  fig.show()\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEnCAYAAACpNTSTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA9CklEQVR4nO3dd5ycdbX48c/ZlrqbuqmbZEMICUmAAEuTFnox0hQviCAXNSpNUAQUf4IFG15FvIKCBe8VFASlJICh90ASSEICJATY9N7L9jm/P87M3clm6u70Oe/Xa1/JzDwzz3c3m+c833aOqCrOOedcSbYb4JxzLjd4QHDOOQd4QHDOORfkAcE55xzgAcE551yQBwTnnHOABwSXYSKiCXzVd/EclwY/p7YT7723q+dPJRF5QURe6MT7pojILSLi/8ddwsT3IbhMEpEjOzz1L2A+cEvYc02q+nYXzlENjAHeVtWmJN87BqjqyvlTKRQMVHVKku+7BbgZKFfV1pQ3zBWksmw3wBUXVZ0V/lhEmoCNHZ/vcEwpdvOS0IVNVTcAGzrZvg878z7nCoF3J13OCQ733CoiN4rIx0AzcICIdBeRX4nIQhHZKSJrReRxERnf4f17DRmJSL2I/FVELhCR90Rkl4jMEZFjOrx3jyEjEakNftZXROQHIrJGRLYGz1vT4b09ReQuEdkkIjtE5F8i8ong+y9N4Pu+QETeF5EmEVkkIudGOCbuzyCsdwDQEhqKC3v9+yLylohsE5GNIvJchJ6bK0LeQ3C56lLgI+A6YBewGugGVAI/AtYA/YHLgVkiMl5V18b5zGOBccD/AxqBHwLTRaRWVbfGee+3gdeAy4BBwH8B9wHHhx1zN3A+Nvw1BzgpeExcInIycD8wA/gmUA38GigHFocdmsjP4A9ADfBF4BigrcPphgO/AlYCvYDPAy+JSJ2qLkikva5Aqap/+VfWvoB64K8dnlMsAPSI895SoCewA7g27PlLg59R2+E8W4B+Yc/VBY/7XNhz9wL1YY9rg8e82OHc1wWfHxZ8PA4IANd3OO6O4HGXxvleXgXeBUrCnjsi+N4XOvEzuCX43rIEfoZlWND5dbZ/H/wru18+ZORy1VOq2tDxSRH5rIi8ISJbgVas99AbuyDH87qqbgl7/E7wz5EJvHdGh8cd33sEIMA/Ohz3ULwPDs6RHAY8pKqB0POq+gYWyDoe35WfASJysog8LyKbgu9vAfZL9P2ucHlAcLlqTccnRORTwAPAe8DnsIvwYdgEcvcEPnNz+ANtX4GU9HuBju8dGvxzfYfj1iXw2QOxoaFIx+7xXFd/BiJyCPAEsBMbUjoy+P75ibzfFTafQ3C5KtJ66AuApap6aegJESnHxtGzLRTABgEfhz0/OIH3bsTu0iMdOxhYFva4qz+DT2O9gvNUtSXsM/oBWxP8DFegvIfg8klP7GIW7mJsHDzb3sCC2Pkdnu/4eC+q2gbMBj4TvpFMRI7A5jDCJfozCPVgekR4fxthAVdETiSxYTNX4LyH4PLJU8A5IvIrYDpwKHA1OXBnq6qLReR+4IfBi/pc4ETgU8FDAlHfbG4GZgKPiMjvsVVG3wc6rpxK9GfwbvDPb4rIk0Cbqs4Jvv8a4F4R+TM2d/D/gFWJf7euUHkPweWTe4Bbgf8AHgc+iV1wt2WzUWGmAX8Crsd2YE8Ergi+FrONqvoMcBE2sftP4FvYhXtxh0MT/RlMB+7ElqS+jvVAUNV/YwHk6OAxlwGXAEuT+D5dgfLUFc6lkYh8C/gZtgR2ebbb41wsPmTkXIqIyFRgEjAPGyI6Ftuv8KAHA5cPPCA4lzo7gHOAG7EdwKuwjWk3x3iPcznDh4ycc84BPqnsnHMuyAOCc845wAOCc865IA8IzjnnAA8IzjnngjwgOOecAzwgOOecC/KA4JxzDvCA4JxzLsgDgnPOOcADgnPOuSAPCM455wAPCM4554LyOv31wIEDtba2NtvNcM65vDJ37tyNqlrd8fm8Dgi1tbXMmTMn281wzrm8IiLLIj3vQ0bOOecADwjO5afWVmhuBi9w5VIor4eMnCsqqvDuuzBzJixYYI/794czz4SjjoJevbLdQpfnPCA4lw9U4R//gOnToXdvGDECSkpg1y743/+FZ5+Fb33LAoRzneRDRs7lg9deg8cfh1GjYNAgCwZgvYLRo2HzZvjNb3wIyXWJBwTncl0gAI89ZoGgtDTyMUOGQH09LF2a0aa5wuJDRs5FogrLlsHzz8PChfbc+PFwwgkwZgyIZK4tq1fDhg0wcmT0Y0SgrAxmz4axYzPXNldQPCA411EgAA89BDNmQEWFjcuLwJw58MorFhQuvjj63XqqNTQkFoAqKmD79vS3xxUsDwjOdfT00zZeX1u750V/6FALFs89B1VVcN556W+Lqs0PrF4NLS123gEDIgeIpiYYODD9bXIFywOCc+Gam+HRR2H48Mg9gJISW+Hz5JNw2mnpXeq5fDn84Q82dLV6tc0RdOsGPXvCIYfsefFXtb0JRxyRvva4gueTys6Fe/99G6Lp3j36MeXldrcemltIhxUr4NZbrXdQWwuf+IQNCfXubb2UV16B9evtWFU7fvJkqKmJ/bmqsGWLvbehIX3td3nJewjOhdu5M7HjRNI3Xq9qewtKS9t7AYMH2wV/wQJ7XFZmcxp1dbYXYcIEmDYt+lxDIGATzjNmWPAoKbGvY4+1ns7gwen5Xlxe8YDgXLhYPYNwqjZ0kw6rV8OSJbbnINzo0RYgQkNIW7ZAjx7wla/AxIkWJCIJBCzAPPOMzT+MHGmBo7UVXn7Z9jjccIN9vitqOTNkJCIjROR5EXlPRBaJyNez3SZXhMaNswtrS0v0Y9ra7IK6//7pacPq1Xb3Huluv7ISJk2CU0+FI4+0Pw86KHowAJg1y4LB6NHQt2/755aV2VxJt27w61/bpLQrajkTEIBW4Juquj9wJHCFiEzIcptcsenVC046CVaujLzrV9VeO/ro/EgToWrpLqqr23c3d9SvH2zd2j4c5YpWzgQEVV2jqm8F/74DeA8Ynt1WuaJ03nm2iuejj+xCqWpf27fDxx9bL+Kii9J3/poaG+aJl4ZCNfZmNbANbWvWWM8ill694M03k2unKzg5OYcgIrXAwcAbWW6KK0YVFXDFFTZp++STNmYvYvsQpk2Dww+3Y9Jl6FAbjqqvjz7Zu3Wr3fWPGxf7s1paog8/hSsrg927O9NaV0ByLiCISG/gYeAaVd1rGYeITAOmAYyMd3fkXGeVldkY/RFHtNcd6NYtcykrLr4YfvQjWLfOchiFzhtaNrprF9x4Y/RhoJCqKntPW1vsndW7d1sgckUtpwKCiJRjweA+Vf1npGNU9W7gboC6ujpP7VjsWlrgnXfg3/+2O/myMhvuOeGE1KyaEbFAEMm2bfDGGzb2HgjAfvvZfoFBg7p+3mHD4LvfhXvvtRVH4QGhpgauvhr23Tf+51RW2tLU+fOjX/BV7ed4zDFdb7fLa6I5ki5XRAT4C7BZVa9J5D11dXXqNZWL2O7dlvL53XftTrhPH7swb9xoK2Y+8xmYOjU9d/Uvvwx/+YvdeVdV2Tl27LDzn3UWnHNOas6rCqtW2d4BVctqOnp0cp+9YgV8//v28+k4lxBK4nfooXDllZlN2ueyRkTmqmpdx+dzqYdwNHAx8I6IzAs+9x1VfSJ7TXI57Y9/hMWLbSdv+IVs2DBbY//gg7Zu/6ijUnvet96Ce+6x84TvW+jb18778MPWqzjzzK6fS8R6BPF2IMcyYgRcdx3ccYftfK6stJ7Uzp0WOA89FL78ZQ8GLncCgqq+AvhvpEvMqlUwd65t3op0ISsrs6GbRx6xeYB4Y+2JCgTg73+3Cd1Im9jKymzlzyOPwJQp6du8lqzx4+G22+xn9sYb0NgIBxwAxx0H++zjwcABORQQnEvK3Lk2SRrrQlZZacMhK1bsveu3sz76yJZyxvq8igqbiF6wwCamc0WvXhYAjjsu2y1xOSpn9iE4l5TNmxNb+hmqO5wqW7cmdjddWmqBw7k84j0E1zltbbBoETz1lJVtFLF8OqecYsMT6R6C6NPH7sLjUbV8P6lSXp5Y3eJAIPrqJOdylAcEl7zmZvj97y17ZlWVbZ5Shffes81cJ50En/98eiuKHXqo1S1QjR58du609BKpGi4CG28vLbXJ42j5g0I7m9OV68i5NPEhI5e8Bx6wC38o+2ZpqV0cBw+2FT/PPms9h3QaMcImRVeujPx6W5vl/D/rrNRNKIPNSxx3nE1qR+sprF1rvaSurAxyLgs8ILjkbNtmhedDKZQ7KimxDJrTp6c3e6aIpZEYOdLyC23b1r4jd906m0yeOtXy/afa+efDmDF23vAiM83Ndt7KSl/G6fKSDxm55CxcGD8NQrdutqxx8WI48MDox6m258/p2TP5C2hlpaVvCM85VFJihWROPdXy/KTjotyjh63rf+456wlt2GDnKS+3vQennmpzHM7lGQ8ILjnbtyc2NyASPVlaUxO8/jo88UT7SpzBg+GTn7Q9A8kkjuvWzVJRH320TeSKZObOvHv39ov/xo0W3Pr3z+xE8tq18Oqr1lMpL7eUHYcemjt7H1ze8YDgklNVZRfeeKJVFNu9G26/3WoXV1fbhK+qpX24+24r5nLVVYlXLguXyrmCRJWVWTqJTGprs81xzzxj33OozvK8eXD//fDVr1rRHOeS5HMILjkTJ9pFqK0t+jFNTXanvN9+e792//3wwQc2IR3KqyNigWaffSwv0YMPpqftheIf/7BkfjU1Nrner5+Vxhw1yoLD7bfbUmDnkuQBwSWnb184/vj2ZGsdBQK2AufMM/e+y9+82YY4amoiD+uI2IT0iy+mr4B9vtu82YLBqFGRh+569bKvf/wj821zec+HjFzyLrjALkxvv22Tp/362fObNtnQz/HH23xAR++/b0Ek1hxEWZkFlcWL4bDD0tP+fBaqahbrZzhggKXMXrcueoGdrmhrs57cK69YbYY+fSx19sSJsWs7u5zn/3oued262Tj/O+/Y6p4PP7S7+wkTbJJ1woTI4/mNjYnt8gUv+B7N6tXx51dELGBs2ZL6gLBpkw1JrVhhq626d7e9IHPmWL2Fa65JTT0IlxUeEFznlJXBwQfbV+giH291T6huQCLi1QAuVj162C7peAIBW3mUSg0N8ItfWO+wtnbv19ets4yqt9xiw1Yu7/gcguu6RJd6TpzYngk0msZGu+h52ofIDjrIqpvFEvoZjhiR2nPPnQtr1kSvvDZ4sC0jfsNLoecrDwguc3r0gE99yoYbIq1Sam21Cemzz05vEft8Nm6cLXPduDHy66p20T7ttNT/DP/97/b5omgGDkx/2hKXNh4QXGZ98pNwxhkWFFatsn0Ju3bZ31eutGBw6qnZbmXuKi21+Ruwn2H48NGuXVav4aCD4PTTU3/utWttWWssvXpZLyFHSvO65PgcgsuskhK48ELbWfzCC7byCGxl0nHH2TCH5wCKbfhwuPlm2+n98svtF9+qKrjkEqvUluj8wfr1MH++rQ7r08fSfgwYEPnYbt0sAMVa4dTaaj0T/zfMS6J5HMnr6up0zpw52W6Gc9mze7etJiottZ3fiaYc370b7r3XUpiDva+tzS7kxx4Ln/vc3mk4/vY3y2QbK4vrqlUW7C+9tDPfjcsQEZmrqnUdn/cegnP5rGfP5HMXNTe372YeMWLPJcKBgG0M3LEDrrhizwBz/PHw9NM2aR1p6Wtzs014n3hip74Vl30+h+BcsZk714bqOgYDsMejRtm+gnff3fO1YcPgssts0nr9+vacVqo2b7ByJXzhC5aS3OUl7yE4V2yeeMIys0Yb5xexfSAzZ1oRonDHHGMbz6ZPt42JJSUWECZMsAUDvlw4r3lAcK6YtLXZ6qR4ZUX79rUVS5Hstx984xuWb2rXLhuy8voPBcEDgnPFJLSJMFYtarDX46UTr6qyr0SoWtqN5cttqGnIEMtu66uRcooHBOeKSUmJDe/U19uqpGg2bkxdcsE1a2xF0+LFewaAoUNtNVKkNOkuK3xS2blic9pptoooWqGjtjZLLpiK1UJr18Ktt1rPYNSo9q+RI2246ac/bd+L4rLOA4JzxWbSJNu8Vl9vS0jD7d5tz0+dakM6XXX//bZZbfDgPXsHIpYGo08fq5QXq+CSy5icGjISkT8BU4H1qjop2+1xriCVlNjy0CFDYMYMy1IaulhXVtrS0ilTYo/v79gBixbZn7172zBUx4nl9ethwYLYy1D79LEA9P77lvzQZVVOBQTgXuC/gf/JcjucK2ylpVbV7uSTrZ5FQ4PlIRozJnaRm9ZWePhh26AWyqMUKno0ZYoVTwol1Vu50oJPvMnpkhL4+GMPCDkgpwKCqr4kIrXZbodzRaOiIvG9A4EA/OEP8NprdtcfHjja2uC556yAzlVXtb+WSGocX2mUM/JuDkFEponIHBGZs2HDhmw3x7ni8d578PrrMHr03r2I0lKbLH7rLUuWB7azWTV+UGhr893NOSLvAoKq3q2qdapaVx1r2ZzrurY2WwkSq6CNKx5PP23DSrF2OPfrZ2VVweYoJkywuYRoduywTXATJqS8uS55OTVk5HLE5s2WmvqZZ2xsGaxU5mmnWYEWV5wWL7aUF7H062dzEoGAzQ1cdBH86EeW62jgwD2Dyfbt9rt23XWx5y1cxvi/gtvTihXws5/Z8sPBgy1vTSBgq0DmzoXPftYmI33ct/gk828eOnb4cLjpJpt7qK/f8/UBA+Bb3/LJ5BySUwFBRP4GTAEGishK4GZV/WN2W1VEmprgV7+y/6zhY7olJRYYWlrggQdsrHiSrwouOhMnwsKFdqMQzaZNMH78nsGjpsYK+ixbtmfqiv32i78CyWVUTgUEVb0w220oavPnWxe+tjby6+Xltm58xgwPCMXo5JPhzTfbh4M6UrVhoEjlO0Xs9yra75bLCR6eXbtXXrGNSbH0729jydu2ZaZNLnfst5+ls6iv33uhwY4dMG+eXfCHDctC41wq5FQPwWXZzp3xa/GGsmU2NWWmTS53iMDnP283BTNm2O9AU5MFiE2bbL5g+XK44QY45BA4//zYw0su53hAcO2qqy0zZayUxqGcM716ZaZNLreUlsKnPgWnnAJvvAF33WXzS8cd115WMxCwlBXvvw/f+Y73GPKIDxm5dscd177MNJr16+Hwwz0gFLvu3W2IqKoKJk/es8ZySYkFgUAA7rknsd3KLid4QHDtxo+Hffe1HDSR7NplK43OOCOz7XK5Z8MGCwhDhkQ/prrachQtW5axZrmuSTogiEgvESlNR2NclpWWWh6aESPsP/LGjZYeeccO+0+9bRtcc42nGXDtewpiLRsNLT2NVorT5Zy4cwgiUgJcAFwEHAY0Ad1EZAPwBHC3qn6Q1la6zOnTx8Z9Fy2yVAXr1tlwwGc/C0ceaTtRnUu0foFIe1ZUl/MSmVR+HngG+DawUFUDACLSHzgB+KmI/EtV/5q+ZrqMKi+3ceHJk7PdEperqqvbE9fFq80ca1jJ5ZREAsLJqtrS8UlV3Qw8DDwsInHWKrq8sX69pTeur7f8MpMn2xLCnj2z3TKXS/bZxyaOt22z5HSR7Nplk86euC5vxA0IqtoiIuOB4cAbqroz9JqInK6qT0UKGC7PBALw0EPwxBM2Lty7tz03Z44NGV1+ORx4YLZb6XJFaE/Cz39uNw69e+/5ekODDTeG10ZwOS/upLKIXA08ClwFLBSRs8Ne/nG6GuYy7JFH4PHHbUJ55EjbfDRwoO08ray0HEcf+FSRCzNhAnzzm7bwoL4eVq+2fSz19dZzuPxyOOywbLfSJSGR0P1l4FBV3RmsZvaQiNSq6q8BT3lZCLZuhenTLWldaYQFZL162R3fgw/ahLNnOnUhBxwAv/ylJb1bssR6lfvss/feBJcXEgkIpaFhIlWtF5EpWFAYhQeEwjB3rk3+xeraDxgAS5fC2rUwdGjm2uZyX0WFzTMdcki2W+K6KJF9CGtFZHLoQTA4TAUGAgekqV0uk1auhG7dYh8jYnMLW7Zkpk3OuYxLJCBcAqwNf0JVW1X1EuC4tLTKZVaPHomtK4/Xi3DO5bW4AUFVV6rq2iivvZr6JrmMO+AAS0kRS1OTDQ2MGpWZNjnnMs5zGTmrkzxkiKWqiETVVpCcemr8oSXnXN7ygOBsbuCqq+zvK1fumWpg927LazRhAnzyk9lpn3MuIxIeEBYRwfIZ7aOqPxCRkcAQVX0zba1zmTN8OHzve1b45NVX29MSVFbCBRdY+cSKimy30jmXRqIJ5ioXkbuAAHCiqu4vIv2AmaqatZ0ndXV1OmfOnGydvnDt2mUVsEpLreKVTyS7OFRtT9ratdbhHDHCViq73CQic1W1ruPzyfxPP0JVDxGRtwFUdYuI+C1jIerVywvguIQtWwb33Wcb2UN7FlXh0EPhwgs9MOSTZAJCS7AOggKISDXWY3DOFamPPoKf/tQS5I4c2R4QQlU0P/oIbrrJg0K+SGZS+Q7gX8AgEbkVeAXPZeRc0QoE4He/s20s1dV7ZjQJVdHcsQPuvz97bXTJSaiHEJxQfgmYC5yEpaw4R1XfS2PbnHM5bMkSy5ZeWwuoUtbWBKq0lnX/v+gwdCi89ZZNSXkvIfclFBBUVUXkEVU9FHg/zW1yzuWBDz+EMloZtvItxi59gr5brXbyzl6D+WDsGayqORLKuiECy5d7QMgHyQwZzRIRz2XrnAMg0NTClPfv4vDZ/02Phi1s6zOSbX1HURpo4ZC3/8gnXvsF5c27AJtkdrkvmYBwAvC6iHwoIgtE5B0RWZDKxojI6SKyWESWisiNqfxs51xqTfzwUWo3zmFr39E09uj3f8NEzd0q2dp3NP22fMiB8/+HQMBWL7vUCASguTk9QTaZVUZnpP707YIrmH4LnAKsBGaLyGOq+m46z+uc64SGBkYtnsniqhpKW2TvPYsibK+qYeCHb3LQQZ9h2LDqrDSzkNTXw7PPwuuvWzKByko47TQ45pjoVUyTlXBAUNVlqTllVIcDS1X1IwAR+TtwNuABwblcs2QJpW0tHFRXzhtvWOegvENl9YbGEsrblAsOXISVUXGd9cILcO+9lixgyBDbK9rQAA8/DDNnwvXXQ01N18+TTOqK70V6XlV/0PVmAFazeUXY45XAESn6bOdcKjU2Ara09IgjYN482+AevjGtZ0844KASqip3Za+dBeCDD+DPf7afdXhuyR49LPnwpk1WtO7HP+56kbpkhozC/1W7Y0VyUrnsNFL1tb1GyURkGjANYOTIkSk8vct1gYAlXW1qsu7yoEHZblERq6z8v78OG2ZzBOvXw/btFhT69bOS3LIsYA9cpz3xhCUOiJZoeMAA2y0+f74F565IZsjov8Ifi8gvgMe6dvo9rARGhD2uAVZHaMfdwN1guYxSeH6XowIBePllePxx2LzZLjiBAIwdC+edB+PHZ7uFRWjsWOjd28YtevSgtNT2HOxRXbW11fJhHeCFFTtr92670McbDqqshJde6npA6Er6657APl07/R5mA2NFZHQwR9IFpDbguDykCn/9K/zhD/b3kSMtcdrIkZZM7Sc/gVmzst3KIlRebtF49eo906WHBAK2+eC00/boTbjkNDTYnyVxrtQVFdY766pk5hDeoX0IpxSoBn7Y9SYYVW0VkSuBfwc//0+quihVn+/y07x58PTTMHr0nv8pRKyr3LOnBYv99oP+/bPWzOI0ZYrV2H7sMQsQ/fvbP8yWLTbHcPzxFjSKkCosXWqrgt5+2yrU1tZafJw8ee8J+Gh69rQfaVubdbaiaWy0DPZdlcwcwtSwv7cC61Q1wq1B56nqE8ATqfxMl79UrTxDv37R75B69LCb0ddeg6lTIx/j0kQEzj0X6upsGcw777SnOT3xRBgzZs8ER0VCFR56CKZPb8/zVFJicyy/+Y3Vmrr6arvYx9Ojhw0DzZ3bYTiug507LT53VTIB4XJVvSH8CRH5WcfnnEuVhgZLjxBv7UC/fvDmmx4QskLE/oEuuSTbLckZL71knaba2j3v6vv3t9/VJUvgT3+CK69M7PNOP92GRXfvjhxE1qyx3sGkSV1vezJzCKdEeC6tm9VccWtttetNvJvM0lLbuelctrW1waOP2l6BSEM8IjYHNmeOXcgTMXKkVbjdssWmZXbvhpYW2LrVNqv17w/XXpv4MFQscQOCiHwtOH8wLpiyIvT1MfBO15vgXGS9elmXObjkPaodO+L3IpzLhPp6u1DHqi8Vusl5663EP/fgg20BxTnn2BDp1q3W25g2DW6+2Zb4pkIiQ0b3A08CPwHC8wvtUNXNqWmGc3srLYVTT4VHHrENOJGo2h3TiSdmtGnORbR7d2LHVVTYRT0ZAwfCWWfZV7rE7SGo6jZVrVfVC4HtwGBgFDBJRI5LX9Ocs4myfv1sQq4jVduQM3myrTJyLtsSmSgGG/JJVf6hVEpm2emXgK9jG8bmAUcCrwN+b+bSpk8fuOEGuOMO645XVNhXQ4ON19bVwZe+FH+dtnOZUFtrv7O7dkUfNlK1YZ9DDslo0xKSzCqjrwOHAbNU9QQRGQ98Pz3Ncq7d4MHwgx/Y6ozZs22J3aBBcOSRtrqiCFc2uhxVWgpnn22riEaN2ntiWRVWrLAbmVjLSLMlmYDQqKqNIoKIdFPV90VkXNpa5lyY0lLYf3/7ci6XHX+8DXHOmGGLIgYOtB7stm22Umj//eGyy7LdysiSCQgrRaQv8AjwtIhsIUKuIeecK2YicP75Nrf1zDO2276tzXoMn/98cjuVMy2Z5HbnBv96i4g8D/QBnkpLq5xzLo+J2EKH0GIH1fwY2kx4Kk7M50Xke6r6IjaxPDldDXPOuUKRD8EAktupfCdwFHBh8PEOrOSlc865ApDMHMIRqnqIiLwNoKpbgmmqnXPOFYBkeggtIlJKMAW2iFQDgbS0yjnnXMYl00O4A/gXMEhEbgU+A3w3La1Kt7Y2G9Tz3UxFZ9Mm2LjR/ulramxZoHPOxA0IIvK/qnoxMBC4HjgJq398jqqmsqZyerW2WqWKJ5+Ejz6ygLDvvnDGGXDQQbGrT7i8t3IlPPywLQEsKbFVH2VllgPprLMSTzmQa3butPXtZWXtefed6yxRjV2WWETexdJcPwZMwYLB/8lmgru6ujqdM2dO/AObmuDOOy0g9OvXnkRkyxb7OvJISxuYq4uDs0XV8u3Om2cpRQcOtOIn1dXZbllSPv4YfvpTuwcYNKj9otncbCmIR42C66/Pr6CwerVtfAqVD1W1f5ZPfhKOPtrvb1xsIjJXVes6Pp/IkNHvsP0G+wBzwz8Tm09IZV3l9HjgAatUPXr0nuu/QhUr3nzT/jd99rPZa2Ou2boV7roLFi+2K2hFhV1BH3gAjjnGdth065btVsbV1ga//S10727/1OEqKiwYLFtmOewvvDDyZ+SaDz+En//cgsCwYe0X/x074J57LMXHf/6nBwWXvESynd6hqvtjNY73Cfsaraq5Hwy2b4cXX7SqFJEWA4vYYPLTT1tGKmc/h9tus6G1UaOs2MCQIe0V7l96Ce6+2zJ05bj33rM5g47BINywYfD884mnLs6mpib49a9t7iM8GIDVsh892n7dX3kle210+SuRAjkCoKpfi3dMTnr33fgVqsvLbY7h/fcz165c9uqrNugeKXNcSYmldJwzx25Fc9yiRfFHAsvL7Vdk+fLMtKkr5s+3nkC01MklJTYsNmNGXsRrl2MSmYJ6XkSuEpE9alKJSIWInCgifwG+kJ7mpUAyt30NDelrR74IBGzifdCg6MeI2C3qs89mrl2d1NKS+ERrW1t625IKs2fHn+uorLReUaQaEs7FksgcwunAZcDfRGQ0sBXoDpQCM4Ffqeq8dDWwy3r3TvzYWHXvikVjo80fRCtRFtKnjxUoyHE1NfHrLataMMiHufLGRltRFI+IBUPnkpHIHEKjqt6pqkdjldJOAg5R1VGq+uWcDgYAEyfa/6BY/zuam22G0XMrt99Ox1l9RiCQF7OWdXXWzFhBYcMGmDAhdqeoM1pbbcw/3o8yGcOHx5/qamuzc+ZiRS6X25JatayqLaq6RlW3pqk9qderF5x+ulWliDSoGgjYePnUqbYUpdh16wZjxthy3Fg2b87Nkk8d9O4Nn/60/fNHuifYts2CRaoWmKnatNWvfgVf/jJ89avwzW/CzJm2Z6Crjj7aAk2sILNuHRx+uA0dOZeMZHYq569zzrFhkJdftgHY0JKTzZtt3uCkk+DMM7PZwtwhYpv1br/dbjEjDcA3N1sgPfbYTLeuU04/3f58+GG7e+7e3Zrf1NReorO2tuvnUYWHHoLHH7dANGKE/fh27YK//c2mXK6/HgYM6Pw5amrgqKPgtdeszR3n/Ldvt3ZMndqlb8UVqbgb03JZwhvTwK4AS5bY8tLFi+25CRPg5JNh7Nj8yU+bCYEA/PnPtn5x6ND2/A6qtsRlwwa45BI45ZTstjNJ27fb4qjly20UceJEmDQpdfsRX3/d9j/W1kYeTVu71sqB3nxz13YUNzdbicZZs6ztvXpZr2HnTusVXH21/Uo7F020jWmdCggiMkRV16akZV2QVEBwyWlrs3JPM2bYlUbEAsWQITYGU7fX71JRU4XvfMd6HVVV0Y+rr4dvfxvGdbH4bKg2b2iFcEUFHHGEVePykU8XT1d2KkfyBJCyAWQROR+4BdgfOFxV/SqfbaWlcNppNpxWX2/LWyorbXOa96b2snq1jd2PGBH7uIoK2xjf1YAgYv8UI0fGP9a5RHU2IKT6irAQOA/4fYo/13XR9t1lbCnfl9Lu1jko81gQUWOjXaTjxcqKChu6ci4XdTYg3JPKRoSypubyhudis3q1TY6+8Ub7Ra53b5tvPukkzwPYUe/eNqIWr3ZuY6PlCHQuF3U2IPwzpa1IgohMA6YBjPT+clqEsoOq2rr30ATp7t1w//2WH+jKKz0ohBs0CPbZx+bb+/ePfIyqTf4eeWRm2+Zcojq71uGJZN8gIs+IyMIIX2cn8zmqereq1qlqXXU+bC3NMy0tcMcdkZOn9expydPeftvW1afa1q22Uue55yxFQz4kmwsRgXPPte+hqWnv10OTwAce6OP+LndlbA5BVU/u5LlcBi1aZHvSoq3LF7GVqE89BaeemppeQmOjrdN/+eX2fEIi9tlnnmkFbPJgUzSTJsEXvwh/+YstK+3f39q9Y4d97b8/fOUrPifvcldOzCG43PHWW/GXLfboYUMjq1fHT3kUT0sL/OY3FohGjNjzwt/SAv/8J6xaZa/Nnm3PjRxplc7Gjct8hbDdu60NPXtGDobHH2/tevllm39parLgesop7VlUnMtVnfr1VNU7U9kIETkX+A1QDcwQkXmqeloqz+ES09SU2N24iI2Hd9Xs2fDOO3vXLgK74JaVWf7/yZMt+JSUwMKFtnRz8mT42tfSv+4+lI7iySctcInYaqEpU2yCvePI5ZAhcP759uVcPsmJCqyq+i9VrVHVbqo62INB9tTUxM8CHgjYV6yiM4lQhSeesFQOkYZR1q+3HkvPnjas1Lu3/X3wYLvrnj8f/vjHrrUhkTY++ij87Ge2HSO09r9/f9v0fvPNeZH01bmE5ERAcLnjyCPb00FHs2GD3Z1HW02TqOZmGw6KtLNX1eoVde/ent8/nIj1GGbPts9IlwULLAfSyJG2XDQUuCoqbBirvNzSPjU2pq8NzmVKIhXT+ifw1TcDbXUZUF1t493LlkUOCtu32xj6ueem5nzRMqfs2mW5B2MNB4nY8NYbb6SmLZHaNn265fiLNvbfr59lTJ03Lz1tcC6TEplDWB38irU2ohTwxXQ5KLTccdMmu3jW1sbOtQPtqaBnzrT39OxpwWH3bhu2uf761CydDBW537Zt79z9zc3tO38bGmwoK5Lu3dNXGWzbNli6NP732ru3ZR/1/QUu3yUSEN5T1YNjHSAib6eoPUWtrc02fb34ol3Aq6rgmGNs7XpFRfKft3ixbSRbtqx9NY6Ifeb550cvJldWBp/7nE2Yvv66ZQetqLDyBwcdZCUTUiGUafvOOy0Ndfg8QmhiOxCwn0u0ZbAtLckVxUtGc7P93OItEy0vj1+0Jhe0tdkw3Pz5FmSHD7e6CV0d+nOFI5GAcFSKjnExbNliY9HLltkdeejOd948G8a59lrbKJaoBQvgl7+0C+2oUe0XtdZWWxL50Udw442xq4YOHmylJNKprg4OPtguUjU17UMzVVV2oQ1VM4tW/aulJX2JV0OBpq0t9sqr3bu7nqwu3VautA2H69dbcC8rs0ypDz5oOQw/85n82Ovh0iuhEprRXhOR/4x3jIuvqQn+679gzRq7Ex40yC6IAwfa49274ec/tyGMRDQ2wu9+Z4GkX78973DLymwIZOVKGx/PtrIyuOIKm7dYu9YC4rJl1isZMsQ2wU2YEPkufd06CyL77ZeetvXsacNA69ZFP0bVft7HH5+eNqTC+vXwk59Yr6C21m4sBg2y34Phwy3D+QMPZLuVLhd0dZXR91PSiiI3b55doKP1AKqrLRi8+mpinzd/vgWRWHf/Q4daioh4S0wzoaICLrrIyk5ecYXt9v361+0idemltqxz06b2CeiGhvae1FVXpXdz2hlnWA8hUvlLVft322+/9AWlVJgxw246IlVqKy21HuTMmRaQXXGLO2QkIguivQQMTm1zitPTT9vQTiyDBlm6iDPOiD+mvXBhe5GzaCoqbPhozRpLypYLKiv3Hv65+GI44ADbr7B0qX3vvXpZjZ7jjos/Qd5VNTXwjW/YbupNm2zoqrzcAsSuXRYI0h2UumLXLruRGDIk+jGBgC3r/fnP4ROfgPHjLc2GDyEVn0TmEAYDpwEdq64L8FrKW1SE1q2LXxC9Rw/r+re0xJ9gbmtLPF9OIJDYcdkiYnMMkydbz6CtzXoGmbxYTZxoF8vZs+GVV2yIaNw4m3TP9Qvn5s3Wk4m0bFbVel8LF1qPcuNGW1Y8Y4b1Jr72NRgzJuNNdlmUSECYDvRW1XkdXxCRF1LdoGLUrZvdrce60Le12V1oIrlwxoyx1UGxtLXZBWHQoOTami0iFgiyparKAsBJJ2WvDZ0RK1jV11vm2qoqO65fv/blvVu32u7s737Xs7MWk0Qmlb+oqq9Eee1zqW9S8Tn6aBuOiGX9equZm8jQxGGH2XHNzdGPWbsWjjoq/UMuLrsGD7beZ8dU4i0t1jOoqrKbjOZmm1cK6dvXgsSDD2a0uS7LcnTks7gcc4zdAUfL/9/cbMMUid6dVlXZ5rLly/fOza9qSzm7dYOzk6pE4fJRaamlEF+7ds9d4WvWWC8xFAzKyvYMCGC9x4ULY6+ycoUlkUnlt1T1kK4e46KrroavfhV++1u7UA8aZHf4qjauu2OHTa4mM5576ql2MfjHP+w/fGlpe4nHmhobH86X4aJ0UrUKca++ahe+nj2t5zRpUuFUhDvhBLuwz59vF/3u3W3VWkmJTTqHqrh1/H5DqUE2bLCehit8otGSyYQOEGkAPoh1CNBHVTM+0lhXV6dz5szJ9GnT5uOPbTXN3Ln2nzEQsDX4U6fa5GVnNDTYhWDNGvsPv//+tqrIi7RYj+z3v7efT3m5BYOWFnt+wAC45hpLYFcImpvh3/+2lWqhZbsrVtj8wIQJ0XcrL18O111nx7jCISJzVXWvLZ2JBIRESqC0qerKzjauswotIITs3GkXpe7dfYw/XQIB28m9aJFdFDsGyNC+h+9/P/L6/XzV3GyBYNEiuO++2DcaoWXJt98efxWcyy/RAkIik8rLwr+AL2JF7g8HKoLPZzwYFLLevdt3K7v0WLLECvMMHWrBt7FxzzH2AQPs+WefzV4b06GiwoYezzzThg63bo1+7OrVcOyxHgyKSdKTyqr6PeAOYAfwaRHxcpouKYGAjV1ns4bAo4/acMhTT8Ezz9hwyosv2kUwFBiGDLHd3C0t2WtnupSVweWX27/D+vV7BsPW1vbUIZ/+dPba6DIv4RKaInI7cK2adcBTwS/nErJ9O7z0kqVJ2LHDnhs3znZfH3hg5uY1PvwQ/v53u/D169c+gd/QALNmwdixNqlcUWHBYOfOrleHy0Vjxtg+gwcesCy74RlxjzvOgoH3DopLMjWVdwKPicgFqrpLRE4FblbVo9PUNldANmyw3b4bN9pwWP/+dhFetQp+8Qs4/XRLuZ3uoNDYaGPi3bvb+cMvgj162CqvDz6w9g0bZr2ZQlltFMmoUVbfYt066ymUlNicigeC4pRwQFDV74rI54AXRKQJ2AXcmLaWuYIRCFjq5R077AIUImJj9X372tDNyJE2Zp1Ob71l7dh3X5tD6FiRraTEAsOSJfbavvvGThJYKAYP9qWlLok5BBE5CfgyFgiqgatV9eV0NcwVjiVLLCtotARrpaXWa3j88fTnVpo1y+5+a2rsvJF2c3fvbpOt69fb5Ksv0XXFIplJ5ZuA/6eqU4DPAA+IyIlpaZUrKLNnxx92qay04aTVq9PblsZGm1Dt1s1SfDQ02BxBaFJV1XZ379plG7omT05ve5zLJQkHBFU9MZTTSFXfAc4AfpSuhrnCsXNnYiVAS0rSv/Jo6ND2cpdDhtjkaXW1TXhv22Z/lpXZ+vyLL/begSsuyUwq70FV1wSHkZyLqbo6/oVe1XLrpHsy89hj4YUX7HwitnroiCOsfaEUH1u3WiqHYpg7cC5cl5LbqWoO1Ntyue6II2yJZ6xN8Vu22DLIdOdXGjMGDjrI1tmHtye0K7ytzR6ffnp62+FcLvJspy7tamrg0EP3vgiHNDbaXfl556V/iEbEEglOmmT5fNassWGizZvtcVMTfOtbVmvYuWLT6SGjVBKR24BPAc3Ah8B/qurWrDbKpYwIfOlLcOedsGCBDcVUVtrd+ObN9vpXvpK5BGo9e8K119oGtdDu5B49bJjo0EOzW4jHuWyKm9wuI42wTW7PqWqriPwMQFVviPe+Qk1uV6ja2mDxYssPtHy5rTw67DCrB1Fdne3WOVc8oiW3y4kegqrODHs4C1vW6gpMaan1AjyVsnO5KRfnEC4Dnsx2I5xzrthkrIcgIs8Akfaq3qSqjwaPuQloBe6L8TnTsPTbjPTq3845lzIZCwiqenKs10XkC8BU4CSNMbGhqncDd4PNIaS0kc45V8RyYg5BRE4HbgCOV9Uopeadc86lU67MIfw3UAk8LSLzROR32W6Qc84Vm5zoIajqvtlug3POFbucCAjOhGr79uhhX845l0keEHLA0qXw5JNWvEXEvg4/3PLphBeUSYeGBnj7baivt30C48fDxImW8dM5V1z8v32Wvfwy/OEPli5hxAhLAd3WZsFh1iy48kpLp5Bqqnbu++6z/D3dullxmieftApmX/ua1Tt2zhWPXJlULkorVsCf/mS1ewcPbq/vW1pqefurq+Guu6xyV6q9+ircc4+lf66ttfMNH25/Lymx+scffpj68zrncpcHhCx67rn26l2R9Oxpd/KvvJLa8zY1Wc9g2LC9awqDpYHu2RP+9rfYKaudc4XFA0KWhC708QqbDxyY+oCwYIHNHcSauO7f33oIq1al9tzOudzlASFLAgFoabHhoVjKy231USqtXBl/0jg0ub1hQ2rP7ZzLXR4QsqS01MbvG+LUnNu1K/WpocvLLSAlosR/Q5wrGv7fPYtOPTX+HfiWLXDaaak979ix8ecGQqUk073s1TmXOzwgZNHRR0OfPrBxY+TX1661OYZULzsdOxaGDIFNm6Ifs3q1VRDr2ze153bO5S4PCFlUVQXXX2+rjOrr7QK9c6cFiPp6uxhfd13qdy2XlNg+g5YWCzrhw0etrbYcduBA+I//SO15nXO5LSdKaHZWoZTQbGqC+fOtvu+2bbbCZ8oUOOAAG+9Pl1Wr4MEHbdVRSYkNI5WUwCc+AZ/+tPVenHOFJ1oJTQ8Ijg0bbPNbSYltTquqynaLnHPplNM1lV12VVd7kXvnnM8hOOecC/KA4JxzDvCA4JxzLsgDgnPOOcADgnPOuSAPCM455wAPCM4554I8IDjnnAM8IDjnnAvygOCccw7wgOCccy7IA4JzzjnAA4JzzrmgnAgIIvJDEVkgIvNEZKaIDMt2m5xzrtjkREAAblPVA1V1MjAd+F6W2+Occ0UnJwKCqm4Pe9gLyN+qPc45l6dypkCOiNwKXAJsA06Icdw0YBrAyJEjM9M455wrAhkroSkizwBDIrx0k6o+Gnbct4HuqnpzvM/0EprOOZe8rJfQVNWTEzz0fmAGEDcgOOecS52cmEMQkbFhD88C3s9WW5xzrljlyhzCT0VkHBAAlgFfzXJ7nHOu6OREQFDVT2e7Dc4lY9s2WLYMAgGoroZhw0Ak261yrmtyIiA4ly+2bYMHHoBZs9qfCwRg7Fi48ELYZ5/stc25rvKA4FyCtm2DH/8YNm6E4cOhtNSeV4U1a+DWW+GGG2C//bLbTuc6KycmlZ3LBw8/DBs2wIgR7cEAbKho4ECoqoK77oLW1uy10bmu8IDgXAJ27IBXX7W5gmj69IGtW+HddzPWLOdSygOCcwlYscL+LIszyFpWBu/7ommXpzwgOJeAQCCx40SgrS29bXEuXTwgOJeAwYMtKMQLDC0tMGpUZtrkXKp5QHAuAdXVcMABNqkcTWMjVFTAwQdnrl3OpZIHBOcS9NnPWg9h8+a9X2tshFWr4KKLoEePzLfNuVTwgOBcgmpq4Nvftgv+xx/D8uWwciXU19vqomnT4Nhjs91K5zrPN6Y5l4TaWvjJT2DJEvjgA5szqKmBAw+E7t2z3TrnusYDgnNJKimB8ePty7lC4kNGzjnnAA8IzjnngjwgOOecAzJYUzkdRGQDVlAn3EBgYxaakymF/v1B4X+P/v3lt0L4/kapanXHJ/M6IEQiInMiFY8uFIX+/UHhf4/+/eW3Qv7+fMjIOecc4AHBOedcUCEGhLuz3YA0K/TvDwr/e/TvL78V7PdXcHMIzjnnOqcQewjOOec6oSADgoj8UEQWiMg8EZkpIjEKH+YfEblNRN4Pfo//EpG+2W5TKonI+SKySEQCIlIwqzlE5HQRWSwiS0Xkxmy3J9VE5E8isl5EFma7LekgIiNE5HkReS/4+/n1bLcp1QoyIAC3qeqBqjoZmA58L8vtSbWngUmqeiCwBPh2ltuTaguB84CXst2QVBGRUuC3wBnABOBCEZmQ3Val3L3A6dluRBq1At9U1f2BI4ErCu3fsCADgqpuD3vYCyioiRJVnamqrcGHs4CabLYn1VT1PVVdnO12pNjhwFJV/UhVm4G/A2dnuU0ppaovARGqRRQGVV2jqm8F/74DeA8Ynt1WpVbBZjsVkVuBS4BtwAlZbk46XQY8kO1GuLiGAyvCHq8EjshSW1wXiUgtcDDwRpabklJ5GxBE5BlgSISXblLVR1X1JuAmEfk2cCVwc0Yb2EXxvr/gMTdh3dj7Mtm2VEjk+yswEuG5guq5FgsR6Q08DFzTYTQi7+VtQFDVkxM89H5gBnkWEOJ9fyLyBWAqcJLm4drhJP79CsVKYETY4xpgdZba4jpJRMqxYHCfqv4z2+1JtYKcQxCRsWEPzwLez1Zb0kFETgduAM5S1d3Zbo9LyGxgrIiMFpEK4ALgsSy3ySVBRAT4I/Ceqv4y2+1Jh4LcmCYiDwPjgACWDfWrqroqu61KHRFZCnQDNgWfmqWqX81ik1JKRM4FfgNUA1uBeap6WlYblQIiciZwO1AK/ElVb81ui1JLRP4GTMGyga4DblbVP2a1USkkIscALwPvYNcWgO+o6hPZa1VqFWRAcM45l7yCHDJyzjmXPA8IzjnnAA8IzjnngjwgOOecAzwgOOecC/KA4JxzDvCA4JxzLsgDgisIIlIrIg0iMi/sub3qD4hIj2CdjGYRGdjFc/YQkReDqa0RkauDufKTyi0lIn1F5PKutCWBc+xVq0BEKkTkJRHJ2xQ2LrU8ILhC8mGwBkbU+gOq2hA8JhV5hC4D/qmqbcHHlwNnqupFSX5O3+B7kyIm0f/D99KhVkEwDfezwH8ke25XmDwguLwRrFZ1SvDvPxKRO2Icnon6AxcBocyzvwP2AR4TkWtF5PMi8mawN/L7sF7EIyIyN1hxa1rwc34KjAkee1uwtxN+J3+diNwS/HttsBdyJ/AWMCLaucLFqFXwSPD7cM4DgssrN2MpzS/CctFfG+PYSPUHUlbMJJigbh9VrQcI5pJajdXeeAq76z462Btpo/2ie5mqHgrUAVeLyADgRoK9G1X9VgKnHwf8j6oeDPSMca5ELAQOS+J4V8B87NDlDVV9KZhx8hvAlNBQjYj8EMtCGS7d9QcGYon3IjkJOBSYbc2lB7A++NrVweR9YOmwxwJrkzz3MlWdlcC54lLVtuB8SmWwCpgrYh4QXN4QkQOAocDG0MVLRIYQ+fc46foDInIF8OXgwzOBc8Mfq2r4+xuA7tE+CviLqu5R61pEpgAnA0ep6m4ReSHKZ7SyZ++94zG74p0rSd2Axi683xUIHzJyeUFEhmKV4c4GdolIKB32wcC8CG9Juv6Aqv42OGwzWVVXd3zc4dgtQKmIRLqgPwt8RkQGBdveX0RGAX2ALcFgMB4r1A6wA6gMe/86YJCIDBCRblghpGiinSshwSGrDarakuh7XOHygOBynoj0BP4JfFNV3wN+CNwSfHkyEQKCqrZipVP/jRVDf1BVF6W4aTOBYyKc+13gu8BMEVkAPI31bJ4CyoLP/RCYFTx+E/CqiCwUkduCF+cfYPV6pxOjwFOMc+0hWKvgdWCciKwUkS8GXzoBKJh8/q5rvB6Cy2si8kdsWGckMF1VJyX4vnqgTlU3duHcBwPfUNWLO/sZ2SYi/wS+raqLs90Wl33eQ3B5TVW/qKoBbHVNn/CNaZGENqYB5bRXversud8Gno+0zDMfBIfSHvFg4EK8h+Cccw7wHoJzzrkgDwjOOecADwjOOeeCPCA455wDPCA455wL8oDgnHMO8IDgnHMuyAOCc845AP4/znfK4PpkDfYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_dataset(suptitle, features, labels):\n",
    "    # prepare the plot\n",
    "    fig, ax = pylab.subplots(1, 1)\n",
    "    #pylab.subplots_adjust(bottom=0.2, wspace=0.4)\n",
    "    fig.suptitle(suptitle, fontsize = 16)\n",
    "    ax.set_xlabel('$x_i[0]$ -- (feature 1)')\n",
    "    ax.set_ylabel('$x_i[1]$ -- (feature 2)')\n",
    "\n",
    "    colors = ['r' if l>0 else 'b' for l in labels]\n",
    "    ax.scatter(features[:, 0], features[:, 1], marker='o', c=colors, s=100, alpha = 0.5)\n",
    "    fig.show()\n",
    "\n",
    "plot_dataset('Training data', train_x, train_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Perceptron\n",
    "\n",
    "Since perceptron is a binary classifier, for each input vector $x$ the output of our perceptron would be either +1 or -1, depending on the class. The output will be computed using the formula\n",
    "\n",
    "$$y(\\mathbf{x}) = f(\\mathbf{w}^{\\mathrm{T}}\\mathbf{x})$$\n",
    "\n",
    "where $\\mathbf{w}$ is a weight vector, $f$ is a step activation function:\n",
    "$$\n",
    "f(x) = \\begin{cases}\n",
    "         +1 & x \\geq 0 \\\\\n",
    "         -1 & x < 0\n",
    "       \\end{cases} \\\\\n",
    "$$\n",
    "\n",
    "However, a generic linear model should also have a bias, i.e. ideally we should compute $y$ as $y=f(\\mathbf{w}^{\\mathrm{T}}\\mathbf{x})+\\mathbf{b}$. To simplify our model, we can get rid of this bias term by adding one more dimension to our input features, which always equals to 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.92180622  0.91789985  1.        ]\n",
      " [-1.06435513  1.49764717  1.        ]\n",
      " [ 0.32839951  2.25677919  1.        ]]\n"
     ]
    }
   ],
   "source": [
    "pos_examples = np.array([ [t[0], t[1], 1] for i,t in enumerate(train_x) \n",
    "                          if train_labels[i]>0])\n",
    "neg_examples = np.array([ [t[0], t[1], 1] for i,t in enumerate(train_x) \n",
    "                          if train_labels[i]<0])\n",
    "print(pos_examples[0:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training Algorithm\n",
    "\n",
    "In order to train the perceptron, we need to find out weights $\\mathbf{w}$ that will minimize the error. The error is defined using **perceptron criteria**:\n",
    "\n",
    "$$E(\\mathbf{w}) = -\\sum_{n \\in \\mathcal{M}}\\mathbf{w}^{\\mathrm{T}}\\mathbf{x}_{n}t_{n}$$\n",
    " \n",
    "  * $t_{n} \\in \\{-1, +1\\}$ for negative and positive training samples, respectively\n",
    "  * $\\mathcal{M}$ - a set of wrongly classified examples\n",
    "  \n",
    "We will use the process of **graident descent**. Starting with some initial random weights $\\mathbf{w}^{(0)}$, we will adjust weights on each step of the training using the gradient of $E$:\n",
    "\n",
    "$$\\mathbf{w}^{\\tau + 1}=\\mathbf{w}^{\\tau} - \\eta \\nabla E(\\mathbf{w}) = \\mathbf{w}^{\\tau} + \\eta \\mathbf{x}_{n} t_{n}$$\n",
    "\n",
    "where $\\eta$ is a **learning rate**, and $\\tau\\in\\mathbb{N}$ - number of iteration.\n",
    "\n",
    "Let's define this algorithm in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def train(positive_examples, negative_examples, num_iterations = 100):\n",
    "    num_dims = positive_examples.shape[1]\n",
    "    \n",
    "    # Initialize weights. \n",
    "    # We initialize with 0 for simplicity, but random initialization is also a good idea\n",
    "    weights = np.zeros((num_dims,1)) \n",
    "    \n",
    "    pos_count = positive_examples.shape[0]\n",
    "    neg_count = negative_examples.shape[0]\n",
    "    \n",
    "    report_frequency = 10\n",
    "    \n",
    "    for i in range(num_iterations):\n",
    "        # Pick one positive and one negative example\n",
    "        pos = random.choice(positive_examples)\n",
    "        neg = random.choice(negative_examples)\n",
    "\n",
    "        z = np.dot(pos, weights)   \n",
    "        if z < 0: # positive example was classified as negative\n",
    "            weights = weights + pos.reshape(weights.shape)\n",
    "\n",
    "        z  = np.dot(neg, weights)\n",
    "        if z >= 0: # negative example was classified as positive\n",
    "            weights = weights - neg.reshape(weights.shape)\n",
    "            \n",
    "        # Periodically, print out the current accuracy on all examples \n",
    "        if i % report_frequency == 0:             \n",
    "            pos_out = np.dot(positive_examples, weights)\n",
    "            neg_out = np.dot(negative_examples, weights)        \n",
    "            pos_correct = (pos_out >= 0).sum() / float(pos_count)\n",
    "            neg_correct = (neg_out < 0).sum() / float(neg_count)\n",
    "            print(\"Iteration={}, pos correct={}, neg correct={}\".format(i,pos_correct,neg_correct))\n",
    "\n",
    "    return weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run the training on our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration=0, pos correct=0.2631578947368421, neg correct=0.6190476190476191\n",
      "Iteration=10, pos correct=0.8947368421052632, neg correct=0.8571428571428571\n",
      "Iteration=20, pos correct=0.8421052631578947, neg correct=1.0\n",
      "Iteration=30, pos correct=0.8947368421052632, neg correct=0.9523809523809523\n",
      "Iteration=40, pos correct=0.8947368421052632, neg correct=0.9523809523809523\n",
      "Iteration=50, pos correct=0.9473684210526315, neg correct=0.9047619047619048\n",
      "Iteration=60, pos correct=0.8947368421052632, neg correct=0.9523809523809523\n",
      "Iteration=70, pos correct=0.8947368421052632, neg correct=0.9047619047619048\n",
      "Iteration=80, pos correct=0.8947368421052632, neg correct=0.6190476190476191\n",
      "Iteration=90, pos correct=0.8421052631578947, neg correct=1.0\n",
      "[[-0.66042328  4.90850882 -1.        ]]\n"
     ]
    }
   ],
   "source": [
    "wts = train(pos_examples,neg_examples)\n",
    "print(wts.transpose())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, initial accuracy is around 50%, but it quickly increases to higher values close to 90%.\n",
    "\n",
    "Let's visualize how classes are separated. Our classification function looks like $\\mathbf{w}^Tx$, and it is greater than 0 for one class, and is below 0 for another. Thus, class separation line is defined by $\\mathbf{w}^Tx = 0$. Since we have only two dimensions $x_0$ and $x_1$, the equation for the line would be $w_0x_0+w_1x_1+w_2 = 0$ (remember that we have explicitly defined an extra dimension $x_2=1$). Let's plot this line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def plot_boundary(positive_examples, negative_examples, weights):\n",
    "    if np.isclose(weights[1], 0):\n",
    "        if np.isclose(weights[0], 0):\n",
    "            x = y = np.array([-6, 6], dtype = 'float32')\n",
    "        else:\n",
    "            y = np.array([-6, 6], dtype='float32')\n",
    "            x = -(weights[1] * y + weights[2])/weights[0]\n",
    "    else:\n",
    "        x = np.array([-6, 6], dtype='float32')\n",
    "        y = -(weights[0] * x + weights[2])/weights[1]\n",
    "\n",
    "    pylab.xlim(-6, 6)\n",
    "    pylab.ylim(-6, 6)                      \n",
    "    pylab.plot(positive_examples[:,0], positive_examples[:,1], 'bo')\n",
    "    pylab.plot(negative_examples[:,0], negative_examples[:,1], 'ro')\n",
    "    pylab.plot(x, y, 'g', linewidth=2.0)\n",
    "    pylab.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAVa0lEQVR4nO3db4xcV3nH8d+zaxt711gJ9q4R2eyaSECgEApeUlDaqiUEpRAFCbVS2k0UUYkFSiNXAhESixd9sRUqCLAEVbUK4QVeQavwtyilhAKVaEnIGhJSMEUQ2Y4TyK4NUZLdxH92n764M/Z4PXfmztxz59458/1Iq92ZvXPn3LH3N2eee8655u4CAMRjqOwGAADCItgBIDIEOwBEhmAHgMgQ7AAQGYIdACITJNjN7BIzu8fMfm5mh83sTSH2CwDo3KZA+zkg6Zvu/udmtkXSSKD9AgA6ZHknKJnZDkkPS7rCme0EAKUL0WO/QtKypM+Z2WslHZK0z91XGjcys1lJs5I0Ojq698orrwzw1AAwOA4dOnTC3cfabReixz4t6X5J17j7A2Z2QNLT7v6RtMdMT0/74uJirucFgEFjZofcfbrddiFOnh6XdNzdH6jdvkfS6wPsFwDQhdzB7u6/kfSYmb2idte1kn6Wd78AgO6EGhVzm6SF2oiYRyW9K9B+AQAdChLs7v6QpLZ1HwBA8Zh5CgCRIdgBIDIEOwBEhmAHgMgQ7AAQGYIdACJDsANAZAh2AIgMwQ4AkSHYASAyBDsARIZgB4DIEOwAEBmCHQAiQ7ADQGQIdgCIDMEOAJEh2AEgMgQ7AESGYAeAyBDsABAZgh0AIkOwA0BkCHYAiAzBDgCRIdgBIDIEOwBEJliwm9mwmf3YzL4Rap8AgM6F7LHvk3Q44P4AAF0IEuxmNiHp7ZLuCrE/AED3QvXYPyXpQ5LWA+0PANCl3MFuZjdIWnL3Q222mzWzRTNbXF5ezvu0AIAUIXrs10i60cyOSPqipDeb2cGNG7n7vLtPu/v02NhYgKcFADSTO9jd/Q53n3D3PZJukvQdd785d8sAAF1hHDsARGZTyJ25+/ckfS/kPgEAnaHHDgCRIdgBIDIEOwBEhmAHgMgQ7AAQGYIdACJDsANAZAh2AIgMwQ4AkSHYASAyBDsARIZgB4DIEOwAEBmCHQAiQ7ADQGQIdgCIDMEOAJEh2AEgMgQ7AESGYAeAyBDsABAZgh0AIkOwA0BkCHagAwsL0p490tBQ8n1hoewWARfbVHYDgH6xsCDNzkqrq8nto0eT25I0M1Neu4CN6LEjeqF62fv3nw/1utXV5H6gSuixI2ohe9nHjnV2P1AWeuyIWohedr3H797895OTXTcPKETuYDezy83su2Z22Mx+amb7QjQMCCFvL7ve4z96tPnvR0akubnz23JiFVUQosd+VtIH3P2Vkt4o6f1m9qoA+wVyS+tNZ+1lN+vx101NSfPzSUmn8Q3A/XzJh3BHGXIHu7v/2t1/VPv5GUmHJV2Wd79ACHNzSa+6UWMvu520nr2ZdOTI+To9J1ZRJUFr7Ga2R9LrJD3Q5HezZrZoZovLy8shnxYDKGvZY2Ym6VVPTSVhPDUl3XprErhZSiZZe/ycWEWluHuQL0nbJR2S9M522+7du9eBbh086D4y4p4UPZKvkZHk/tCPzbr91NSF29S/pqbyHi1wnqRFz5DHQXrsZrZZ0pckLbj7l0PsE0iTp+zR6WOb9fjrdfVGeUs+QEghRsWYpM9KOuzun8jfJMQu7+iRPGWPbh47M5PU09fXL6yrb9wmyxsA0AshJihdI+kWSY+Y2UO1++5093sD7BuRCTFhaHKy+fDDLCNd8jy2nZkZghzVEGJUzPfd3dz9Knf//doXoY6mQoweyVP2oGSCQcDMU/RUllJIu1JNnrJHiJIJE5FQdeZp86QLND097YuLiz1/XpRvz57mpZCpqaR+vbFUIyU96qrUq6vePsTNzA65+3S77eixo6falUKqPtEnb/vo7aMXCHb0VLtSSNUn+uRpH8sOoFcoxaBS2pVqypanfVU/NlQfpRj0paqPWsnSvrRyS9U/jSAeXGgDldK4qNaxY8n48rm56pyYbNe+VuP0ixxDDzSiFAME1KrcMjfHiBrkQykGKEGrckurE8eMlkFIlGKAgNqVW5otOxDyuqyARI8dCKrdydVmPfOqj91Hedxdz5x6Rr/67a/0g8d+kPlx9NiBgFqdXE3rmaddeo/RMnFaW1/TidUTWlpZ0tLKkp5ceTL5/mzyfWm14eeVJT139rmOn4NgBwJLW+UxrWc+PCytrV28PaNl+sfqmdULwrge1s2C+8TqCbmyD1rZtmmbdm/frfHRcf1QP8z0GIIdhauXG6o4fLGX0nrga2tJuWbjaJmqjN0fROu+rt8+99vMYb1yZqWj/e/ctvNcWI+Pjmv36O4Lvo+Pjp/7/ejmUSWXvZDs3ZZp/wQ7CsWJwfPSTqzWh0Ly5les588+f2HJo0VYn1g9oTVv8jEqxZbhLdo9urtpWF8Q3Nt3a9fILm0aKjZ6CXYUqtWJwUELrmbj2CXp2WeT7ywr0Bl31++e/935YK4FdrOwXlpZ0tOnnu5o/5duvfSCnvP4yHjzXvb23Xrhlhee61VXAcGOQjGN/rz6G9m+fdLJk+fvP3lycD/FbHR67fS5IE7rXTeG9dn1s5n3vXlo80VljrSwHhsd05bhLQUeabGYeYpCsfDVxQbpNXF3PX3q6Qt70S1KIU89/1RH+9/xgh2pJY+N912y9ZJK9aq7kXXmKT12FCptGv0gnxjs908xZ9bO6MTqieZhvWGo3tLKkk6tncq872Eb1tjoWKawHhsd09ZNWws80v5FsKNw27adD/adO6UDBwa75FC1xcDcXc+efjbzuOqTz51sv9MG27dsTz2Z2BjYu0d369Jtl2rImDeZF8GOwjS7jNxznc+1iE4vPsWkTYI5F9g5JsEM2ZB2jezKFNbjo+Ma2TzSfqcIiho7CjNIteROdTO2v1eTYNqNq965baeGh4ZzvgLoRtYaO8GOwgwNJZeA28hMWl/vfXuqpj4JJstQvSeffTLYJJhmdevGSTCoLk6eonRVqyX3Qn0STJawXl5Z7utJMKgu/uVRmBhGxKRNgkkbV93tJJhzveeUcdXjo+Pa8YId9KqRCcGOwlT1MndZJsE0/tztJJjGsN44+mP8W9/X2N9/XFuOPCZN7pDm3iP9xQAPFUJQ1NjR9+qTYJoO1StwEsxFYZ11Ekyz4UJcIw8Z9PTkqZldL+mApGFJd7n7R1ttT7CjnfokmCxD9bqdBNNu9Ef9K/gkGIYLoUs9O3lqZsOSPiPpOknHJT1oZl9395/l3TfikTYJJi2su50EkyWsX7TtRaVOgvGjx9SsT592P9CpEDX2qyX90t0flSQz+6Kkd0gi2CO3tr6mk8+dbD76oxbWjWWRTibBmOxcr7rdUL1+mwTz+PCkJtYu7rE/PjypiRLakxkL6/eNEMF+maTHGm4fl/QHGzcys1lJs5I0GfN4tz63emY187jqUJNgmvWum06CWViQ3rsxWK4O/AoU7/a1Oc1rVqM6X2Nf0YhuX5vTQontaomF9ftKiGBv+qnyojvc5yXNS0mNPcDzIoNmk2DShuoVNQmmHta5JsFEFCz/PTWjdx+V/kH7NaljOqZJ3ak5/c9UhY+DhfX7SohgPy7p8obbE5KeCLBfpGg2CSYtrLudBNNsxMfGn8dGx3o3CSaiYEnG98/oC6vn2z0yIs2XNb4/S4ml35ekHDAh/ioflPQyM3uppMcl3STprwLsd2C4u556/qlMQ/VCTYJJC+7KToKJKFgqNb4/6yehQZxG3MdCDXd8m6RPKRnueLe7t+x7DMJwx9Nrp7W8spwprJdXlnVm/UzmfW8a2nTxScSRi0d/1Nes7ucrwZzDEMFiZH1dGXtfCT1dK8bd75V0b4h9VVWzSTCtTjJ2Mwkmy1C93aO7o7gSTMdiWJ+girJ+EqrUxwy0M9BLCrSbBLOxp91Xk2BiQ7AUo5MSy8wMr3efiCrY3V0rZ1Yyj6sOMQkmbVx12ZNgokSwhMcnoShVPtg3ToK5aD2QgifB1AN7bGRMo1tGCzxSoAR8EopSKcG+7us68tSRTOOqO50Es3XT1gvWrO54EgwwaPgkFJ1SVne0l5jrPdm337lt54WljpGUMdZ5J8H0uYGf8T3wLwBiV+krKJmZJnZMZArrXSO7tHl4cxnN7CtVnpjZk7zt5QvAGwgqjvXYI1HVYd49G/7cqxeA8dwoERezHjBVvXB0z95wevUCVPUdFAMha7AzHi8SaTO7y57x3bOVALp9ARYWkrAeGkq+L7RZXzGipQ0QL4I9EnNzSUWgURWGI/fsDaebF6BeVjl6NOnt1+vyrcK9qu+gQAOCPRIzM0mZd2oqqT5MTVWj7NuzN5xuXoBWK0amqeo7KNCAGjsK9/2/WdCe+f16ydoxPTE8qSOzc/rDf6rAicZu6/KMikFJOHmKaqjyKBJOhKLPcPIU1dBNuaOu0xObnaKsgkgR7ChWt6NImp3YvOWWpEzSbchvfKOQqnliAsip8ouAoc91e+WdZj39etmwm1mlaTNT5+cpuyA69NgRRlrZpNtyR7sefdZyTl2ekhDQZwh2tJalzt1qPHi34zCzjAvvZFJQLyYWFX1OAMjK3Xv+tXfvXkcfOHjQfWTEPYnr5GtkJLm/0dTUhdvUv6amwj53nv0X0cZ27W32WgE5SFr0DBlLjx3pspYviugNN/b0paS336jT0StFj4BJe61uvpneO3qOYEe6rIFd1DT7mZnkxKa79PnP5xu9UvTU3FZvYlmWKgACItiRLmtg92I8eD3k19eT7+0CuVm9u9N9dKLdmxgnatFDBDvSZQ3sqi1U083iXnk1e602YgVI9AhLCqC1flwXZdcu6eTJi+8veqmA+mvVbNx+L54f0WNJAYRRZPmimbxDBhcWmoe6VHyPuf5aHTzIUgUoFcGO6ghRQmlVx+7VmulVK01h4FCKQXWEWG0xbSleKelJE67oYz0pxZjZx8zs52b2EzP7ipldkmd/GHAhxsOn9cp37iTUMTDylmLuk/Rqd79K0i8k3ZG/SRhYIcbDp43kOXCg+3YBfSZXsLv7t9z9bO3m/ZIm8jcJAyvEeHjq20C4GruZ/Zukf3H3gym/n5U0K0mTk5N7j6YNCcNg68fhlUCPBLs0npl9W9KLm/xqv7t/rbbNfknTkt7pGd4pOHkKAJ3LGuxtL7Th7m9p80S3SrpB0rVZQh0AUKy8o2Kul3S7pBvdfbXd9ugzrC8O9KW8o2I+LemFku4zs4fM7J8DtAlVkHeyEG8KQGmYoITm8kwW2nh9USkZ3cLoFCAX1opBPnkmC3F9UaBUBDuayzNZqBfXFwWQimBHc3kmCxV1RSUAmRDsaC7PDM5eXFEJQKq249gxwGZmujvZWX8MM0iBUhDsKEa3bwoAcqMUAwCRIdgBIDIEOwBEhmAHgMgQ7AAQGYIdACJDsANAZAh2AIgMwQ4AkSHYMVi4AAgGAEsKYHBsvABI/apQEssfICr02DE4uAAIBgTBjsHBBUAwIAh2DA4uAIIBQbBjcHABEAwIgh2DI89VoYA+wqgYDBYuAIIBQI8dACJDsANAZAh2AIgMwQ4AkQkS7Gb2QTNzM9sVYn8AgO7lDnYzu1zSdZKYvgcAFRCix/5JSR+S5AH2BQDIKVewm9mNkh5394cDtQcAkFPbCUpm9m1JL27yq/2S7pT01ixPZGazkmYlaZK1OQCgMObeXQXFzF4j6T8l1ddBnZD0hKSr3f03rR47PT3ti4uLXT0vAAwqMzvk7tPttut6SQF3f0TSeMMTHpE07e4nut0nACA/xrEDQGSCLQLm7ntC7QsA0D167AAQGYIdACJDsANAZAh2AIgMwQ4AkSHYASAyBDsARIZgB4DIEOwAEBmCHQAiQ7ADQGQIdgCIDMEOAJEh2AEgMgQ7AESGYAeAyBDsABAZgh0AIkOwA0BkCHYAiAzBDgCRIdgBIDIEOwBEhmAHgMgQ7AAQGYIdACJDsANAZAh2AIhM7mA3s9vM7P/M7Kdm9o8hGgUA6N6mPA82sz+V9A5JV7n7KTMbD9MsAEC38vbY3yfpo+5+SpLcfSl/kwAAeeTqsUt6uaQ/MrM5Sc9L+qC7P9hsQzOblTRbu3nKzP4353NX2S5JJ8puRIFiPr6Yj03i+PrdK7Js1DbYzezbkl7c5Ff7a4+/VNIbJb1B0r+a2RXu7hs3dvd5SfO1fS66+3SWBvYjjq9/xXxsEsfX78xsMct2bYPd3d/S4kneJ+nLtSD/oZmtK3nHXM7aUABAWHlr7F+V9GZJMrOXS9qiuD8GAUDl5a2x3y3p7lq9/LSkW5uVYZqYz/m8Vcfx9a+Yj03i+PpdpuOzbDkMAOgXzDwFgMgQ7AAQmVKDPfblCMzsg2bmZrar7LaEZGYfM7Ofm9lPzOwrZnZJ2W0Kwcyur/1//KWZfbjs9oRkZpeb2XfN7HDt721f2W0KzcyGzezHZvaNstsSmpldYmb31P7uDpvZm1ptX1qwb1iO4PckfbysthTBzC6XdJ2kY2W3pQD3SXq1u18l6ReS7ii5PbmZ2bCkz0j6M0mvkvSXZvaqclsV1FlJH3D3VyqZd/L+yI5PkvZJOlx2IwpyQNI33f1KSa9Vm+Mss8ce+3IEn5T0IUnRnZ1292+5+9nazfslTZTZnkCulvRLd3/U3U9L+qKSjkcU3P3X7v6j2s/PKAmGy8ptVThmNiHp7ZLuKrstoZnZDkl/LOmzkuTup939qVaPKTPY68sRPGBm/2VmbyixLUGZ2Y2SHnf3h8tuSw/8taR/L7sRAVwm6bGG28cVUfA1MrM9kl4n6YGSmxLSp5R0pNZLbkcRrlAy6fNztVLTXWY22uoBecextxRqOYIqanNsd0p6a29bFFar43P3r9W22a/kI/5CL9tWEGtyX1/8X+yEmW2X9CVJf+fuT5fdnhDM7AZJS+5+yMz+pOTmFGGTpNdLus3dHzCzA5I+LOkjrR5QmJiXI0g7NjN7jaSXSnrYzKSkTPEjM7va3X/Twybm0urfTpLM7FZJN0i6tl/ejNs4LunyhtsTkp4oqS2FMLPNSkJ9wd2/XHZ7ArpG0o1m9jZJWyXtMLOD7n5zye0K5bik4+5e/4R1j5JgT1VmKearinA5And/xN3H3X2Pu+9R8o/y+n4K9XbM7HpJt0u60d1Xy25PIA9KepmZvdTMtki6SdLXS25TMJb0Mj4r6bC7f6Ls9oTk7ne4+0Tt7+0mSd+JKNRVy47HzKy+suO1kn7W6jGF9tjb6HY5ApTv05JeIOm+2qeS+939veU2KR93P2tmfyvpPyQNS7rb3X9acrNCukbSLZIeMbOHavfd6e73ltckdOA2SQu1Tsejkt7VamOWFACAyDDzFAAiQ7ADQGQIdgCIDMEOAJEh2AEgMgQ7AESGYAeAyPw/SugVn8RyP80AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_boundary(pos_examples,neg_examples,wts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Evaluate on Test Dataset\n",
    "\n",
    "In the beginning, we have put apart some data to the test dataset. Let's see how accurate our classifier is on this test dataset. In order to do this, we also expand the test dataset with an extra dimension, multiply by weights matrix, and make sure that the obtained value is of the same sign as the label (+1 or -1). We then add together all boolean values and divide by the length of test sample, to obtain the accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def accuracy(weights, test_x, test_labels):\n",
    "    res = np.dot(np.c_[test_x,np.ones(len(test_x))],weights)\n",
    "    return (res.reshape(test_labels.shape)*test_labels>=0).sum()/float(len(test_labels))\n",
    "\n",
    "accuracy(wts, test_x, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Observing the training process\n",
    "\n",
    "We have seen before how the accuracy decreases during training. It would be nice to see how the separation line behaves during training. The code below will visualize everything on one graph, and you should be able to move the slider to \"time-travel\" through the training process. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def train_graph(positive_examples, negative_examples, num_iterations = 100):\n",
    "    num_dims = positive_examples.shape[1]\n",
    "    weights = np.zeros((num_dims,1)) # initialize weights\n",
    "    \n",
    "    pos_count = positive_examples.shape[0]\n",
    "    neg_count = negative_examples.shape[0]\n",
    "    \n",
    "    report_frequency = 15;\n",
    "    snapshots = []\n",
    "    \n",
    "    for i in range(num_iterations):\n",
    "        pos = random.choice(positive_examples)\n",
    "        neg = random.choice(negative_examples)\n",
    "\n",
    "        z = np.dot(pos, weights)   \n",
    "        if z < 0:\n",
    "            weights = weights + pos.reshape(weights.shape)\n",
    "\n",
    "        z  = np.dot(neg, weights)\n",
    "        if z >= 0:\n",
    "            weights = weights - neg.reshape(weights.shape)\n",
    "            \n",
    "        if i % report_frequency == 0:             \n",
    "            pos_out = np.dot(positive_examples, weights)\n",
    "            neg_out = np.dot(negative_examples, weights)        \n",
    "            pos_correct = (pos_out >= 0).sum() / float(pos_count)\n",
    "            neg_correct = (neg_out < 0).sum() / float(neg_count)\n",
    "            snapshots.append((np.copy(weights),(pos_correct+neg_correct)/2.0))\n",
    "\n",
    "    return np.array(snapshots)\n",
    "\n",
    "snapshots = train_graph(pos_examples,neg_examples)\n",
    "\n",
    "def plotit(pos_examples,neg_examples,snapshots,step):\n",
    "    fig = pylab.figure(figsize=(10,4))\n",
    "    fig.add_subplot(1, 2, 1)\n",
    "    plot_boundary(pos_examples, neg_examples, snapshots[step][0])\n",
    "    fig.add_subplot(1, 2, 2)\n",
    "    pylab.plot(np.arange(len(snapshots[:,1])), snapshots[:,1])\n",
    "    pylab.ylabel('Accuracy')\n",
    "    pylab.xlabel('Iteration')\n",
    "    pylab.plot(step, snapshots[step,1], \"bo\")\n",
    "    pylab.show()\n",
    "def pl1(step): plotit(pos_examples,neg_examples,snapshots,step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8561af1ae77c421f9ca068fe0bdac566",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='step', max=6), Output()), _dom_classes=('widget-interact…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.pl1(step)>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interact(pl1, step=widgets.IntSlider(value=0, min=0, max=len(snapshots)-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Limitations of the Perceptron\n",
    "\n",
    "As you have seen above, perceptron is a **linear classifier**. It can distinguish between two classes well if they are **linearly separable**, i.e. can be separated by a straight line. Otherwise, perceptron training process will not converge.\n",
    "\n",
    "A most obvious example of a problem that cannot be solved by a perceptron is so-called **XOR problem**. We want our perceptron to learn the XOR boolean function, which has the following truth table:\n",
    "\n",
    "|   | 0 | 1 |\n",
    "|---|---|---|\n",
    "| 0 | 0 | 1 | \n",
    "| 1 | 1 | 0 |\n",
    "\n",
    "Let's try and do that! We will manually populate all positive and negative training samples, and then call our train function defined above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pos_examples_xor = np.array([[1,0,1],[0,1,1]])\n",
    "neg_examples_xor = np.array([[1,1,1],[0,0,1]])\n",
    "\n",
    "snapshots_xor = train_graph(pos_examples_xor,neg_examples_xor,1000)\n",
    "def pl2(step): plotit(pos_examples_xor,neg_examples_xor,snapshots_xor,step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8f45bf78e4c8471fbc6eea233dde2bf6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='step', max=6), Output()), _dom_classes=('widget-interact…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.pl2(step)>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interact(pl2, step=widgets.IntSlider(value=0, min=0, max=len(snapshots)-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As you can see from the graph above, the accuracy never goes above 75%, because it is impossible to draw a straight line in such a way as to get all possible examples right.\n",
    "\n",
    "The XOR problem is a classical example of perceptron limitations, and it was pointed out by Marvin Minsky and Seymour Papert in 1969 in their book [Perceptrons](https://en.wikipedia.org/wiki/Perceptrons_(book)). This observation limited research in the area of neural networks for almost 10 years, even though - and we will see this in the next section of our course - multi-layered perceptrons are perfectly capable of solving such problems.\n",
    "\n",
    "## Complex Example - MNIST\n",
    "\n",
    "Even though perceptron cannot solve XOR problem, it can solve many more complex problems, such as handwritten character recognition.\n",
    "\n",
    "A dataset that is often used when mastering machine learning is called [MNIST](https://en.wikipedia.org/wiki/MNIST_database). It has been created by Modified National Institute of Standards and Technology, and contains a training set of 60000 handwritten digits, collected from around 250 students and employees of the institute. There is also a test dataset of 10000 digits, collected from different individuals.\n",
    "\n",
    "All digits are represented by grayscale images of size 28x28 pixels.\n",
    "\n",
    "> MNIST Dataset is available as a training competition on [Kaggle](https://www.kaggle.com/c/digit-recognizer), a site that hosts machine learning competitions and contests. Once you learn how to classify MNIST digits, you can submit your solution to Kaggle to see how it is rated among other participants. \n",
    "\n",
    "We start by loading MNIST dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# If you are not running this notebook from a cloned repository, you may need to grab the binary dataset file first\n",
    "# !wget https://github.com/microsoft/AI-For-Beginners/blob/main/data/mnist.pkl.gz?raw=true\n",
    "# In this case correct the link to the dataset below as well.\n",
    "\n",
    "with gzip.open('../../data/mnist.pkl.gz', 'rb') as mnist_pickle:\n",
    "    MNIST = pickle.load(mnist_pickle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now plot the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  0   0 188 255  94   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0 191 250 253  93   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0]\n",
      "1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlIAAABRCAYAAAAZ1Ej0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABJF0lEQVR4nO2dd3Rc133nP/eV6R29EyABgmBv6t2qlhzJsuPY2Y1LsnaKnbWTbE682STr3XU2PtmUdbKbeB3HiRU7cZEtybZkqzdSIilS7A0s6L0MBtNn3nt3/xiwSSwAiDKE3uccHBIzr9wv3i2/d+/v/n5CSomNjY2NjY2Njc3MURa7ADY2NjY2NjY21yq2IWVjY2NjY2NjM0tsQ8rGxsbGxsbGZpbYhpSNjY2NjY2NzSyxDSkbGxsbGxsbm1liG1I2NjY2NjY2NrPkqgwpIcT9QojjQoiTQogvzlWhiglb47XPUtcHtsalwlLXuNT1ga3xPYmUclY/gAqcApoAB7AfaJvt9Yrxx9Z47f8sdX22xsUvm63R1mdrXFoaZ/pzNTNS1wEnpZSnpZQ54LvAw1dxvWLE1njts9T1ga1xqbDUNS51fWBrfE+iXcW5NUDPeb/3Ate/8yAhxGeAzwCoqJs9BK7ilguLCy8meQIiIjMkAT7Oe1DjUtE39VEceOydxy0Vje/legpLX+NS0Tf1kd0WsTUWOxmS5GRWXO6YqzGkLnbhd+WbkVJ+Hfg6QEBE5PXifVdxy4VlSPYyxiBtYgs75YvkycF7UONS0Qfwgnx8lPfgMwRbY7Fjt8UCS0Xje7mewrWt8Xx2yheveMzVLO31AnXn/V4L9F/F9YoOJ24ypM//yNZ4jXERfQ6WkD5Y+s8QbI1LAbstLg3eCxpnytUYUm8BzUKIRiGEA/go8OO5KVZxECBMmgRpmUQWDG5b4zXG+fosaQFEWEL6YOk/Q7A1LgXstrg0eC9onCmzXtqTUhpCiM8Bz1Lw4v+mlPLwnJWsCFCEwkq5gb28TpoUwPdtjdcW5+ubavTjS0kfLP1nCLbGpYDdFpcG7wWNM+VqfKSQUj4DPDNHZVkw1BWNWCEvht+Bmsyj9Y1hTcSwksl3HVsqqiilip3yRSbl+J8uSAEVFaGqCFVB+P0Q8hc+FwLL60LJGYixCaRpgbQwxyfAMmd9u0XR+A602hqkxwWASKYx+uZupviMPoAX5OODc3bhImIxnqHQHQiXE6WsBKmpoKkYYQ+mU73gOD2eQ0lkYGAYK5lG5nOzul8x1FOYqqteN5bfheVQMV0aasZASRsoPYPIVBorlZrVtYtF43xht8WlwUJoFLoDxedF+LxIl2PqQ1EYB/0u8gFn4SMpUZN5lEweJZZExiaxEkmkJUFaIN/lvjXnXJUhda3S/psVbLmhnf9e+xP+Zvgutn9rMxVvxmBPcRjVaiSEcDqRXjfjW8sYvqFQEaQuWdnSx4nBMoIvRdAyEi0tCbx6EnN0bJFLPUuEQKgqvb/YwGSLgbAEgeMqlX83jDTNBWkENrNACNTyUvINZZx82IMRMFEDef7zpp/xAd+pCw79g777efV4M42PBXCdGsHo7luwDm5OOb+urs5x46pTbAl28bHAAb4XX8PPh1Yz8f+aCR6egMPHrz19NjZFgtAdqJXlJNdUMbZGJ1k3NVGggHRabF11mq/WP4UqBHFL8tfD72N7fyPZXXVU7M7h2d+DTKWRuRxWJjPv5X1PGVLKhjZGtgapW9fPB0r3U6vqhPQUlgooi5gtRwiEw4FaXYlRFqDrfX5yQYnllKhVKW6oK0SZcKoGtwRPsMe/jJ9l1iAUiZSQCbXg7zNw7+lEJpOzfhteFISC0DQmmw1u3XAMSyq8Ya2kUlXBkiBnP9M23yQ/fD3xGhXHpMTfl0d/Ye+sDQShaSgeD9RVISbiczojN2dMGRJKYz25mhDdt7nIVBusXdVByJEi4khxg7uDctV7wWkPl+zF25bj6Y+txdlXS/X2clxdE5jHTy6SkFmgqChtzUQ3hEhfn+DWhk5UIenORvhRYhUeJccHKg/wlw9WEG2NsExtRRkaxxgcWuyS27xHUcNhRNBPYk0lplMgFTAdAtMhSFUKrKlJHi0NelziGzBxTBg4O0eR8QTm2PiCl1nxeFACfuI3NDBZrxHbkKOsYoTVocJEgYLEq+W4I3SMctUDgE8YPBDeT6UzxnOuVXQ2laDf0UTpfom3L4uy/cBVrdhMh/eUITWyNcjaXzvEr5e/wjqHiS5ULCkQkkV9exSajuLzkmopI7rSwR984vt82NePU1z88dzjPcl1/tO0OfuoVLN8pOKTdB8vpXmoHHVw7JoypIRSMCJXtvbxT/WvAHBvKoDQNDBNCj6pRYgQDD2a5fc2PM8zI2s58tYyml/TsXL5WRl/wuFAREKMbQwTOOVCFKUhpSDcbhKrSxlZp/EXH/8m97vfWdecmO94aB/wTPIBz5t8tfpNfp728DvOT1H+VgneEx3z3sHNCUIgdI3xjWHyvzjO37Q9ySbnOL/V+TAnJsp4Md3Cr7W8waeCR3n0jqN8e9N6ftB1L+EjGtiGVPEgLhMKaKnNHgoB5SWk64J0f0Ci+nNomknIl6bKm+Avlv2Q5ZobgO1ZhZ9MbOTx/ZtwdbmoeqMcV58bxqML/ndRAn7M2jJ63i9Zv/IUP1hxac+hM72MLlTudSe5132YPyg5DGsgYWXZUvtZPPvd1OzSkFnbkLpqtMoKhh5sInpbhk+Vv06TniEh4X8O3cRT+zbQ+moU0T/KgnfpQqD6/eQ3LufEI04qVw3zcNVx7vJ04hSeS55Wqji4y3Mav1BwCgd/uvIJjjTU8r1Vmxnb0UTDUyUFgyqRxEokll4nUQxISfW/Ovi/+x7mM594mlONJQifFxFPzKrRKgE/2cZSJh9JkHk1QMWb81Dmq0BxucjdvJqOT0jWNHTzSEk7W51jgHtG19nqHOPzv/BT/rr+fZS6t1Lyeh9GV8+VT1xEtOoq+h5dRvLGFH/b9iQ/jm7kj4fqcf5zGGfUoMKUfPWX7yF7o87Hgnt5yHeQ9t+s5LUX1rFioBZrZHRBlhdsLkTxelH8PkbvaSJVIUjWmyilWRrKz820mJbCUMyPcdqHIybQE+AesVDzhT7T25tGPdaFlc4g80ZxG/5CoHg85K9byeD1LoxNcdoqu/mv5fvxKxlUYeERWQJKhmr1nB9jm56kJPIGG27qpn9riD33NHB4uBJj/43UvphGeX3vgkmI3tFItFXhrnUHuCl4kpiVwSN0VCF4NhVk3PSRkxoJ08Vo3sdYvjD7/UD4IMu0MVY7CiaNU2g8uOoQOyLL6HZtpnxvHufTb81buZe8IaW4XFjlYcY3WGxa1sN6RxpQ6DXg+c6VeE454PRxrHT6iteac4SC8PuI1zrZvPUEn6jcPvWGf2kjCgqVpEY99+jucOW53nmcmzwn+NXMJ5g4GSSkCtQhDSuZKurlsYuhKyZKKIgVnUAaxmIX55L4dnfhiFfT8OkRakMxqChFkRIzm53ZhYQAXScX1HiwaS9PnbihUG9z+eLouIVAuJwkqh38zuZnuN17nNW6g5kaUQAlipvfCHZxYMVxXm7bQOBUBHV8omgNfsXjwSoLEVuf446mU9zomuBLI7XEDpWw4uXjZ5c/3DffxP7VtXwwsI8GTeO/Vz3L7cuXY1aEUBJJWERDSnG5QNcRLhdYJjKdAcs6kzcNLIk0TYQiQCigXHrmRnE6QdcQbvfUjLGFjCewMtnFratnlp09nkL5XC6skgDZiIexdaDUJrlneTu3Bdv5oHfg7GlZafBWNsjfld/F6WiEyQkPqQEHSq7g6pENeIlY9ajxDCKTQ07GkZnsRTcmLSqKiuL1QGMNY6tcWFsm+f3VL3CP9yQ6oAqBKSXq1Kxc3JLEOde3RhSTu9xd4O7ilwL7eb20gb913slEdxllR0uwJmIL0hfn/IJs2MKpGmSkzmnDgYokLxUeH93CaMaHaSkk8g4m0y7SKScISbpRZ62/D7+yn4ii4FF0bg8cJ6BleHKTg9h4kPJ5LPfSNqQUlfRdaxlZr/MP7/9/NOsxPMLNz1J+nhrbRPVf6zg6OjEWqRNXXE4SG2sZWy/4ccOP8Qmd2T4St3CwxmHy803/QM96J7/07GeJvO2n/N9iWPH43BZ8nrm+pJMnful2qrbFYdfBxS7OJTGGhtEDPiZMD/dVHOFvf+cu6p8M4vrprlldL+9W+I2S13mhbSWZO9bi2dddFD42QlWhNEK6XPDx4DE8wnHV1/xU6TYqH5zkO85bCbWtofyHRzBjk0VlTAlNI33naoY36Hzrrr8jZTl5LNaK9b0ymp89jTEePXusnoTTsRKOlFegOgap19xsqeth/72raHgqD9HoZe40v+RuWs1kg4OxjRaOCYWyvRZ63EDNFAwfNWui9Y8h3U4svwfLpSHVixhTqmBorZtUpaR0yxDjcS+ZUTf1T4Pv8BBGV++iGVNqeRlEggzdWkqiHgIbxriv9jAPBPZTpqbRkagCRkwHL2fOpSrRMQmpKb7R9CMyUmIBGSkwZUF/SmokpYOnopvYF61l4NVWQu0WgR/uLp7NMEKg1VUT31jFuv+yn08EnuUmdw9BRSVlwZ8M3Ec05yZj6iiiUF5LXtxY9mg5Hi17mxotyj+1PcbnfvmjtK9upuXro1inu2e943a6lH33EBVeDyeXt7Jr1Ub+z30JjNM+vL2CqtcnUKJxNMMkZKUIySRSSoSiMFxZx49WtfF399zBpzdv4/dLjnCfZ5itrn5WuIb40xMfsg2p2SCcThSfl/FVOumVWdr0GB5FJWZleHz0TrafWk5r3zjW2PiiNAa1JAIlYcZXaYj6BCXK5d/un0yGOJ6pImp4qHFGuct7jGrVJHzeeRoq5aqHchVWtvRx3KqhbE89Wv8YxsC1s9O4xhElscwkd9jJ1Q/Z84iUiGyO7ZPNhPQ01bXjZIPluISYXZ0S4FcELkce0yVAVa98zjwjnE6UQICRWyuJr8zjE853HWMh6TbSHM6V06yPElIK3gu6EPgVBxrv1lGtpbnZ286Lq1fS5y6l/LUSFNMqHqN/agPI2CqdzMoMrXqSb8ea+Gb7jVT05Aq7ZM97xoEuk+E9ZfxR5mHaygf5q/qnqPNEebM5ixFyI2ZbJ+aAbFgjVSmobx1iMuOk3x9GzTpQpsZEJSdwD/swXYK8FywHSPXdZZUKGA1pIuEkj9TupytdytFwBclgNV51ETfrAKnNDcQadSY25SitmOTh+gOscA6RkTovJesYyIfYP1FLfyLAeKywHKQISSSYZEtZD39ZvY2daT+7k010pyOkTR2AiCNFpTOGUzFYFRxiaKOfEV8AX+8a9M7hxd8UMlVPY1urGV2r8gvht2nWo1SpbvbmLN5Or+DF3WtQkwrC5Fxit0tURanC3to6SsNxbqs8VTC4KrNYHidC1+bdkLLicaxkCl0IwqKSTImfYK+FdyCH6OzDuNjLlqKiqSraMh8OTx6PWlgRcAodv7CIqAmkNr9tb8kaUkooiKwqJXDPIJ9r2EFEdRKzcvQYOm9sb6NyF1hDI4vmu2A21xJb7qHy/h4eqLhy2IUv7n4U/bAH96gk3gid95TyaHg3Nzsv7o39WPP3eau+hC92/Colhz04ryFDarWzl3XrO+nf01TchhRANsdzh9toqBnjgeojPB6pJORwIHO5qxo45eUcYxcQJRTEbKxkza8f4uGSd/tKWEiyMs9T8XX804kbeLRpP+vcBZ+nMm2SNj2JR+jv2jhRo3qocWd539rH2d6i8MfPfAavlFAkhpTQdITfR+DuQb7Y+CphxcW/dW6h9BseXEe6Md6xzOF5YifLnhSoKxrp2drMzi9Vs9nbQcXWSX74zD34HQ7kTJd854h0iUKq3uDrK79Do+bC2lzoMxQULCxiVo43MhVE1ATLtAQRRcMpdCwuPO7M/wEsLEYCWbpKPHwu/LlF0XUWIej6sOTz1z/Dp4JHzxr7fxVt5mtdt9NzpBL3gELNqwnKBqKEu04AU35/N63m+feVkfr4y/zR4UcwtkUoOZJHTxSeb0eNk8kGBccN49xee5LXt/4DP1i1gj/zP0TDT2pwLrIhJTQdxe8j+rEEH2/eze3uFBoeLCR/P3QXr7Y3s+rLXRhDI9ObLRQCNRjAbKnn6TtvIr0mzbKqMcxACbrTCQuxkckyMQYGUQYGqdl27uPLlt6hky5V+GjrHm5wn7rckfPCkjOkFL8fUVvJwJ1lTKw1+E/1r3Oz+xQKDr4R3cQ/HriJyp2S4IExrNz8WteXY2SDl+h6k09Xv8VmVxdQeAOykGzP6HTmS9kea8aSAguBftRDuN3E35EkeMrJq73X8WzJdeTCFh+7Yzt3+w9zsyuPMvXK4RM6y7QoydVZ1IyTikVTurSRhoES1clUaNzrP8i/Bu9C8fsKQVKn65smJRgGSl4yYipk81rRNMzkdcsYXaPxqfAh1jsGOd9/r89MsSNTwx/teQRx0kPpQclPIrfxpKtQB3NByDZmuX/1YX6t9DVW6Vx0J2qdmqDvl3NoJ6qoeS2C6+Twojugy40rGV3l497KbZSpk/z+4PVMHCmhrH0QK564xEkSkcqgZgsGdL02znJ9hO9E7iNUWoIxMLRoS19KWuFDb3+abFbDGHajlOQIBpJIKTAthXTagaJa6LqJplgoioWcWv4RQtIYGqfFN8xvlbxBleoBFLal6/jnvpvw9xowOs6ibbGVEv9hB1/lbv61YiuJtJNsrw9vj4Jn0KJ+1ECfzKN1DhV8Rs/DdCm4hgU37fh1nK/5qX4rgTY8Cbk8AOFhD/5OD5M9IZ6vu45bP9WOS+RYsXKAyR21OBV1UX3D8reuZXS9k4ebtnGf/yAaGtuzCk9FN/HGi2uoOCSxJuPTL6OUWMk0Wvcw1a8pZI47SfqqKTnVg1mku8EVh05mWQnJGsED/gM0aGlm4795NRRLfz0nCE1DiYSIt4SJXZ/hd7e8yKO+o0RUJ1mZ5+WRFsKvugjtGcA43bmoZU0sg1WtvXzAd+rssl5WGiRknm3J1eyL1bKnfRlYhc6s8pSF/3QCDpzAIS2qdnkQAT9WxM/TjavxN2W40XUEZWoZxSk0ytQ8rQ0DnBxoQGha8azpLyWkRI8r5AyVzQ4VwycRLhdCVZEz6GClYaLmJSOml7y5+Et6Z4g2a2TXprjJ3UONes6IspAMmk62TbYQfM5D5HACdh3Ed965Wl0tsetreM7RygZfN3XaiYsaUhWqgy9vfop/rryJ/lgDVfEQdPcual2NN3oZ2yC5038Ev5LhmRNtBE6Bebr7soOSzOVRcxJVWFSoaWo1N7mAQPo8CEUsiq0hVRCmwNoVIjAsKd0fJ9biI1l1brBxnXe8ybvf/vcsD9FRF+Gj4V1UqYUZqT3JZRw/XkNLfxpzIrYQUi5JuN1ATTtIBsvwjEoadkYR/SMXBCp+l6u0omDpAveYhfqSj/IdMeTew+eOO+PArqpETgUJNlZy4KN1LHcOcVPpaX7sr5txO59rxlud5G+e5EOh3axzqGSlwZvJ1fzkxBqq3zDw7OqcsQEk8zmMwSHE4BBuCiZJ8W75AVSVdLlOttRko9NCWWAjCpaQIaV4PMTfv5bhzQq/+/CP2eDqpkFLk5HwZsbJ1wbvoGtnLSu2jWKNLHIUcCGQFNboz/csePj4h+h6qxbXiMA5Llm1YxTMqZ53fKKQemJqjdpMJBHpDEoiQepgG/8qt/Ab4QP4xLlB2CkUbi89QWdzhNSDm/AdHsE82bFwOt8DyFyewGnJSGXgygdfBisaxTVczePjW0mMewjNTfFmz5TvRXxVnk+u3kVQudC4s7DYmVrB633LqX6hB2s8yjttBHNwmMDLKXLelXxl9AMk7nqeu7xHWevQLzjOKTTu8/RR2fAzvvWRm3nbWkvtUC1m/9C8+2S8C0VFcbsYWyN44Ja3adZjPJNYScX33fjao1iXGzSFQPg8ZAMK6x2DVGvv9idbcIRgYpVF8+o+ul+tR09JxNEOwp0uIs7pl+/0f1hGVVscHYuoleaFVC2P791M82NZ1FN9Cx865h14t53A53SAriNzuUJAyczll1KtdBrfaydA0xCqgjUZP+s6pHi9yFWNxJp9RFsVwtcNcVvlAT4d3slfj9zGC4/dQNWu+MLXz3cQX27x5bVP06QZnMwbfOHUR+jYWceyn6TQTnZiRRc+FtSCoqgIl5NsUCAdJifzBhWqRVBxXfncOWRJGFJC0xB+H9EWFb0lxqeDPYDAwMmraQ/bky3sPLScklPA8BjyCg1sXsuqO1C8boygyTLvOCrnfGE6h0qIHAb3WB7neBbz2MlLNwLLRFomVhL8HTDhDvLtlha2uk+z2VEY9HRUVroGqI9EGawP4OnzXvxai4yUkomMm24jRa3mBorDP2haWBZ6ykLkrs7ZVhoGSs5kNOcFRZIJKQTeYXAsJGooBGURAmUJtnhPo7/DYTwvTV6PrmBiIEDlSOdFfQ1lPoc5Nk7wdAbD7eJfGq7jdHUZv1f+IiFFIXBeZxdQXKzUJ3m4ZC+vNrcxuakafyK54NGVFa8HUVdFrsLglkA7PYaHtyYb8Z2OI0auXBapKkhV4FcEGirWpbx6FxDpMan1TtDhrkOqFLbuz2T7vqIirGWUuRI4hMW4Jfjp2HpcPQ60k11Yk5dY6lxAzNnsipTy3HlCoJZE0CrLMUv8ZMNOoiscJOolzpYYj9btY72rm8cmNvPM6dVUHc6iDk8s2kyN0B0oQT9myGC9sw+n0Og33Zw4VkPpcdCOdhbyzZ3x5ZuaXSv8Xzm7DFvM4WUuiaKiRkIQCWFGvKRDThJ14ApnSEoNiwuNW6+SxfRaaI0NWMOjhYDVc2xcLglDSgmHMRsqWPvgMR4pPecQm5cmf9V9L+0H61j1X49iJZKYi1xx1PJSco3l3LjuBH9ZvQ3tvEl1OeCiZPcosquvENdqGg9b5nOUfustKqoq+Wb7Q3zt/hj7rvs2MBWUzBMjU/MGf3RDOcleL57d8yZtVkhLIkyT4dMlfKX0Hr5S9SIXLjQUOVKi5iXCmLtdS5WVEwzeECFyNAiLNIGYW99I750ufm3Fs9zrTqKc11VkpcGIaXDwhZXU7TMK8a4ug7L9AOU7VJQXa9m3fgN/+QfwQHj/u6Kil6seHvTE8N//DZ67aS17f2s9yluTC9vZN9XS8cEwd607wAPefh45+lG6D1bRcuowZrI4fUSmg1vNY1RnSQ+7CM7gvDOpi3Ihiy2BTvyK4JV0NfufbKNqXw5zZGTeyrxgKCqKQ2fifc2MbBTccecBbg0e535vFzkpSUnBE5Mb+F9d96N+MURj9xDmyBjGIqZdUKsriN5YQ13tEHWagi5UtiVW0vrVURgefddSq9B0lFAQoSqF3cCWhcznr70crYqKGg4yeccK+u6W3LyuneuDHejCIKIlcLxjXtwpNOq0STavPs3BX2um8akAysETc77J7Jo2pISmIRwOxu9bznib4CORI7Q5BwAHWWkwbhm091bg7VEKWeeLwPqWHhfZEgchR/rstnALScLKIvICkc1jzdCXSRoGMh4n0Jmnc9TLpJXBpzhRECgIdGEihCzOiR5pIfMG3m6Vl0ubSVY+v9glmhmWhZKVKHmBMUcLHA2BKIEVWfKBShZrTmqywUndLT1c7zl1dgPDGToMk7cyTQROS7wnJ7GuNKBMzZ4yMob/pIvnXtzEnvW1+Ff+kDY9eUEIDwVBkxbjRt9J3nZsnA9pl0UqCqZT4lbzqAiGJ304owrkrxwcVagqE1sqGF8NulAwMIlbOZQ8iLxRyEa/gCgeD0o4hDuUock9AgkdLTmzMgi3GyrLMMMGq519vJSq5ftDW4gcNXD1Ti76kt47EZqGWlpCbkUVqOJdu1+VnImSNbBcOpZDIe/VyAYVklUKibYcKxqGcCoGO+Ir+EbXrYwnPaQmXWiDDlwjgrre0wUjZTGDjyoqRlWYoevhA6U96FPuHHmpIiYTmMlCcGnF40F4vWTX1ZOs1Ik1K5gOidRAWKAlBCVHTLwdCeT+Y8UR/PcKCEUgdJ14jcqt6w/xUMl+lusjmAhcwiSiGJgSolYGc2oMDSmCB0oPEb/RRadZT6BtI6U7RmBiEnNoeE7KdW0bUg4HSsDP6ANZfmfjC3zY33F262tC5uk33DhPuQidMguO1kWA9DhJl6j4tXMWcVbmGbEkigFYViFZ7wyx0hk87SPoo9WMWBJd5HGfFzhRiMVfYrgoUiLzOcLHDUZ1H7EbisfRejpIKdEyJkpeKzTcOfgzbwz0sKaih/8Z+uSiGVLxRsGuVT8BwHyHpoPZan4+tobwwUmsQ8emfU1zIoY4mqH561V0/WI1T5VvoqRkO+F3TObVax50MYipK6higeMTqQJLA00xMZGkxj2EoiDNK88+CE1j6AZYsbYHHZWUlWfIVAq7+K4wazcfKMEARl0pTaUjrHH1oEcVnLEZGlIeN9naEJGySTY6k3x64E72Hl3Gqh0dWJOT81Ty2SMcDszaMvpu9yC1Quyr89FS4JiUZEMCwyvJlRmEKif4QssrbHB1U6Hm+GLvQ+zqaqD0KTdVvRkcHQNYY+NYmUxROF0rDp14rZsP3LabR0Jvnw1JkZcqVjJV8NsSAiUYwKwsoeceB+7WCb69/lvUaXlKFDcJmeXNTIjPvvnLhF8LUHZUx8pR/MaUUMDlJFkr+cf6l6c+PDNmKIBOr5FmxHRgSQWXMFjlUPh4oI+PB/r4n6VreW6glclEGf5TDrANqULHhdtFdekE7/cexXVesMBbtv8moZ97WbY3ijIawyiSCiJyBnpSng34BtBlSL4TvRH3sEBOxGZl9MlcDmtoBF9XNV/qfYjfq36WDVN2VKtjiA+37uVn1bfg83rnZY141kyt3cdrNdKNOTzCZGKxyzQDZC6H41AP/hUr2JV14Vo1QcenGmj8tjbjSM/KZJqdB1eQWaVzd/1h8m6B4vUWXTqKbw/cwPFdy2iZ6J/xwCJzOazBYWpf8PL60PUMfSbAf6v5KfWa+4KZLxXIlOi4K8sxenrntPyXI1vqZvmmHm7ynyAvLfxHdUr3p6fXJhUFSrOsCfWjCsHbWT8/HN+Kd9DCGh1btEEqmnHTmS9DzGIlKru6jtH/mOI3l2/HkpLdpxvwndCRyWQhVloRIZxO8te3MrzRRdv97TR6x6hyXLjElbF0UpYDv5pBERbDuQCHY1V8+bUPoI9pOKOC8AmDhrE8js5eZDpdyFm6SDHA3oUQCJ+XdEThk5HtVGsGeanztYlW3hxuJPcLFcQbFHJrUoQCKSLuGL9VuZ1m5yBNmoFHKQwKLqGx3jHGn13/I77sfz9D2kYqXxjCPHF6kQVeHmmayFicwCn44ImHWBfso9xxoUH/f/bfgeOIBzULphPSjTnuWH2cr9e9wkeCu1nl7uMPHvxFgntDVOybm3Jdu4aUoiKCAXK1EZYFTlGvFbZmG5hkpIFy3Efpk4cxJxOX32mzkAiBdOrk/AK3eu4NNSNVRnJ+lLwsJMaczdq7lFipFO4xi7f7aukrD7HBUXACLVMt7vAf5Sn/LQiHA9KZ4sq/p6pkSqG0chJXMS4/Xg4pMUdGcI828XJiFfWhCU6u1TAjPpQBHSsz/b+zyOVxDmmMLPMSUnKYLhA+byEIXrEYvkBfLIinX8xulmWqnirt3ZSNhTn+4XLGqpzUYp0N3QGF3GDZkMAqCUDvAkUFV1TyXoU7Sk9To0WJWxLPkIU+MIE5nTYpBE5XnnJHHAWFvnyYvaM1eCaNRQn8Ky0LkTUZGg/wWqAFNS1QczPrWzIlGr/T+iJbXV2YSJjQcY3Lqw44Ox8IVSUX0Mj7oNodo9U9QLPz4oGIS5Q0urA4kS9hIBPEOaQROSIJnJhEHO3ASqWKYvbpoqgqli6o1oyptGLQmSkhbymMrxG4VkX5StvTLNdHCCp5/IrAAuJSkjQLBmFI0YioTj7oHWdPw1G+v/Y6Svf5EacWNy7WFZEWMp3GN2hy5HA9XdVhwp4L8+S693mofDONmsph+hwMpdwcKK+COlihOwkpfaxoGKJ7oHbOinVNGlJn1sH7PlhP2y8d5XOVL3Jmeu9k3uDfJq7DPVLYmr5oQeLeiaKiBgN03xfiP37ySe73tnMmuOEah+DPq5/nuoZ1VFWUIfsGZh0FOXAiTuaFIDuaV/CgZx9QSBR7uzuF4ZOFrb6LFM/mYghVRTgcRG4c5PG2xyhV3Zwu2h7s0gT2D/H039yG5yOD/POWf+bzbZ+jJFEL7aemP+DkDVxjkM7p1GtuJhshsLEB1yuTixaBf76wEglkOk08tYoJ0wNc6MjtFArRG3MYnhBVh/X532auqGgVZaTKFR4K7GPCcvN8qgXPiAHDY9N+hkJIlKmpn/3JeoaPldEcSyzK3j1zeAQxPkHLf6tn1FdDQ2IYkZiZgZD3KHzQ14WOSqJYOo1LYKVSeJ89gG+7l5M/WMaxcBuG7+KL43236hiNGT6zfhutvkEyd2oczbTg69bOJvYtSqTEmojhHTL58tAdPBrew82uPH9S8Qr5ckm8TeARkpCioQuVuCX4p9g6jiWqeHuoYDhoqsUfr/wpG5zDVKluPh5+k1V39fN/d3+I8s6y6UdBXwykxMpk8P78AK3bvIVVqXek0vIlD2OlM2BJ9GW1ZO92UfEOY2uuuSYNKSUYYPLGZUyuNPmF0n1Uq1kMnJzMG3w/toXv7LiR+i5jananON6ahCJAVTA8cKv7JBHl3J9eQyWgqEhNIq8yZ5VI53CPW0waFwYl01CRgstmdl8shBD4HVnKzwv4eK0hJ+NEjoXoT3qIqBkmVoKWKcF/qnPamxxkNot3wGRw0kNemoWt6nrxPa85QUqkaZIddfNsbC3Xu7ZfEAMNQHMamDOIdXTVqCpSLTitjhgBjqcqUVPG9JawhEDoGk7dwK8UjN6JvAc9LhA5Y3GCIEz5H8r+IZSpUBpX2mF5FkVFKy8lFxTnklTLPHpMwRmzFtxxfrpYmQzk8iiJJNqoH91z8R3ApaEaEmNuvpa9Hd2bI+RPk64y6b/NS7hiDe6hDMquw0WxQeldmCausTzPHF9DzboJbnQdIay4sZAEpclJw2JbuoqXY62cjpdysqMCZVLDNVoYWzI6HKivp16LUqVChWqxwdlTCBwb8iNGRovmRftSWJkMXOHlUi0twSgPYC1Psy7UN6/luSYNKVlVzsTH4/xK034+5BsF3MSsDI9Fb+b7O6+j9fcOI7PZomsEQtMwnZIWff6294tUBtdonqRZ9FnqlhTm6BhidIzUL1/PiOnmlrsP8nLtSgI/0aZdD62JGKFtXYxsbCRm5Wbl03JNISWhQxo/0jfzuXtfw1cMvdGUfXAoXcsbw42EJjPTmg0UDgfC7abMm6TOUdhSHs25cY4LRHZx+6HZJIJWHDqp9XUka+RZ3zVTSvyd4D8eu/JOzcXEMgtL6pd5br6uHnxCUO10ItesoOvBMlpu7+J3H3iOr/XfUXCoP+LFnEwU3eyMNAyc7YPUfLeWH4Y28NnwQTShYmExZOb4TvQWHj+8kaonHQT2j7CyY98FfZDi9/PcTatY5epntSNKWHETdEgyZZJsVQD9tApFNnbOGEXFaKllZKOHf7j+a6zQJ5nPtDHF0HXNCMXvxwi7aQgPUj/VYeWlyZCp8P1dWwnvVwtGVJHs0juDNE2seAJvv+APhzfxyfAb82pQ2SwuccOJnGGQTmmaWIkkzgnB86llOFZMMiADNO8OIUdGi+7FYC6wdBAOE/UiE2/5qBP/uFy45Xldw9IhqJjUOsap8k6SUsundaridCJ9HsKuCcrUOAoq0awH15hEZIvLKXta6Dqpch0jaGIheTHt4eex6wh05RCDI0Uz06+GglAagVgCmclgJRLTL5uUWLk8as8wtS/pDI028FuNv8Z1Nx7n+jWn2POHbUQOQunL3VijY0W1vG5NxPAedZJ8spL1x79QSAMkQc0I3IOCmk4D35FRGBl791hoWVjyXINLyCzjpokeF+ixzKx2jRcbQhFEWzxMLreo0yYJKfNr6lxbhpQQKKEg2RKd1f4RKvUJoBDq4ES+gtB+ncixTHEOOFNOtp5hi6c7VnOb/xgt+oV+IVID6XYU8jfN9j6aiulU0M6bzrCQZGW+MMOxBBpJUWNBRuqkDAeYl1mWEwKEUog2rExFHRYCoSooOTiYqmNjVR9HHXlkOICSTGEW4XbzWTO1W9Nygu4yeKfJaUmJHlNxTSxMfRWKQGoqlgZeoVCmxal0xzmtTjPdt9OJ9LqIOFKElBzgJplz4I6acA0aUkJVyEQEqr+wFLgt0cILXSupH0gUVRBHEQ6RbirB1aWixJSCITUTLBNzaBhlaJjq7jrytSXkrld5qOwAq+4b5FveWwmdKEHNZCCbLRoD0kom4WQHFeksZTsDSFVFWBYikz8bH+lSUwlCVVGmwuFYWMQtkx7Th54AJZbCLLJJiJlyJohsolag1yQoVVWcU075FhZ5IJ3XEcbcuU1cM4aUGgggIiGOfaGamlVD/HbpK3gERC3Bnbs+g3UgSNPP+5Gj40UXJO4sUuLtyzLxdpAjLTXc7z5xwdcfu2M7T9avo/bPlqMcPlUIUzBdppzZB++ppvSXeviVsu1nv3o57eJP2n+J8DGJFY0Wp6G5RPB1qPxR+yO0hYc4XZYq7JI8vwNWVLSqCmTIT6ohQKJGI10myK5J4fdm0DWT+6vf4LdLtpGXcLy8hN/++K9ScjBC8Ds7FlfcHKKuaCS+pozK+3r448afUKZe6AuVkRbluy2C2zsxFqC+SksiDBPFgKS0yMuZxTMzWmoYW+3ml/ynqVV10jJHPOUiMpRe1JRUs8bpZLLVYH1tPwqC5/paYWcQEeta7JIB5zYc9Xywhts+toc3v7GJ8u0aYnRs1v2b2T+EFpuk7+9X8ZXWZv79Iy/zizftpGdDmJP/0Erp9iGsju6i6j/NoWHE2DhCKbyKWKYJlzGEtJpqjJoS1kXaWescQMHFoVwJ3xu9Dn+PgdXTX3RLmdNCURF6IUB3+rZWxtp0nFvHubG684L0VjuzOs9NbsL653KWnYjNme/iNWNIiXCQfE2EspZRHq3dS63mZNzM0mk4yJ0KUHHIxBocnpnxsQhosTS+XhftyUrGggcoOS+q893+w6jLLZ5ZexthrRntRG8hUfEVNAlNKyx5rqpnsgk+X7ODJi0BeLCQHM9WM3q0lPpho6g6gaWIe1Qy2FXCTeUd1IRjyKZa1EQaYZhYPg+WRydW7yETVEhVCbIRCyuQpyKUwOsozFxYUnDa8NGmJ2nSxzF9FoZrgQNTXoZyX4KO8jDS50E4nTPbYSoEitOJUeZnsl7l1kgXWxw5tPO6osP5HDvSLbiHsgs/+yEK+3+VaTqoCd2BEvAx3uhiolVSp4+Rx2RbJkxm3IUSGy+6eEvTQSgKeihLrWcCgFjCTXBIIotkdk04HFgVEbIlki3+DrarmxGmeVUDo8znMGN5QkcmEVaAp7euZnNpLx8q28MfLm/FPVaKp3+osFRWJDNT0phZn27UlTK+ykuTe4TQVJfSmStl/3A1kUlj1rvFFxOhO1Cal2EGXKRLXQxv0sg1p7mrsod13p5CKBWZJ24ZPD5+Jy91t1B7PI7Se+lZu5lyzRhSybZKRjbq/NcVT/CwdxQFhf1TlnT16yaup/cUT7yoy3Gym7KRCV66u4Xv+Xr5VPD42QjkN7vybHHuQfm85One1ej/vIJA+yTsP3rZSyolEfItNXT8Brx/5W4+5h9CmQqtELMy/HRwLc3/MonoGyne2bolQvjgJErez8rbB7kneJjPfvJTqKkgWlog18VZXTnAn9Q8R5maxi8k34uv4aXRVjp+2oQclDgnTZ5rquXxti18+bYf0eoYINCuEuwong7uvzT+lJdL23j+7VsJmCbmqc5pDyyK04lorGN8tYfkdWk2eztxigu7od869suMb6uksadnQWajYGppz6FjOqBUdeNVpvf3VivKmNxaS+rDMZ7Y8E+s0BR259x87uVfoXy7VghwWCSD7ozQNFqqhtni78BCkou6CHRmkUXiJ6QEA/TfEsbSJH9+6F6qj2Qun+R9ukiJtf8owVM+OFrL8x/exLoP93DLPQfZvbYO3+GrC0+z2HTf7+Oeh97ig/4DZ1MzvRJdSfbNEhxDo9fc+CB0B2plOUc/G2Z5az/faf43XEKgCwUdFVUIFBRO5i1eT63m9X/aSv0rY1jtHZhzGFLlioaUEKIOeAyoBCzg61LKrwohIsD3gGVAJ/ARKeUsUnBfHjUcxlpew9BWHc/WUVr0YVKWyds5P/+7+2463qynqXviqoyojExxmLfIkkEgqKGRetFMXuY4yA7SpMiRQQgRvlqNMpfDSiTxHHDzVeVOmq8bZKU+Rr3mQUHgFDp3+o+Qr1H57g03k/eEKBufChwmz70RCk0tRFJ26AzcV028CR5qfYu7AkfO7rJJyxxPJZZz/LBO8vA/ks0XHDEvps+NB8nC78RRVi5nfEOYm4O7ruo603mG7oJxOa85aNRoHF+/g+OpSioDEyxb008qr5PNa7gdeUbTPv6s+0HShs5owku8O4CnX6V8fw5HNIuSzKLkg2DpnLyuglbHAMICIeWC1lM1A7uyeZq1PAHlwk0RdWqC672n+G7bHQizHM/p7ssGeD0/51eqQmdsrYDGJO9fcZRmxzBQeJHoNFJ8q72Fvb/1Q8TgJL2ZPDWyYd40XrK8CHRM3EoOqSmgXCRI4dQSbWJDDb0PmXys4TCVqsnenIMfRbcQ2qcT6Hh34vHp1tPFaItnUMvKyNeWcGN4N23OvrOBUsU0jZQFaYuKglQKDtbZjI6St+bOYJUSmU6jDI4RPBXifx++i3sbj3F77UkONa3DLSWJ00cXrC3OBWpZGbk1deRWpLkvdJCQojBqpvm3yXXsPN5Ew4E8InqhD+ZC9jdnyxkIIPw+UmuqUXMWjsE4jIxjjk8U2qCiogZ8CK8X6fMQW1dCvE5l07p27i45SoV64c48C4tRM83XR+/m5zvW03gsC8PjSGNuUzZNZ0bKAH5PSvm2EMIP7BFCPA98EnhRSvkVIcQXgS8CfzCnpQMoizB0XYDKW/v4Rst3qFIdDJkWT4xv4fSuepq/0Y81dHUZyAWCZtYREGEMmWcXLxKRFQzQSYRylolWXpNPY2JctcYzCYarX4kzMeDjp80byIUOUq8VnCQVBDc7LerCbzJxm4dn9A0Eugo7h4Qp0eJZpBBIp4qlKRhejbJf7OELNTumZqLOOdClpMn3+7eg9nlYkWst6OPi+jrlMbo5cdEyzyexthATDye5PXDsvLxqMx9EpvMMO+UxxhmunFsFF2KNjOEA9kVrWe/t5r83PXn2uz/teoj2gXIcRz24RiSlJ3LUnuzH6OoBCjvvTcAzVo5ruJzjH6ngocDMNM5VPdXj8LPJ9YRCbxG4SC68MjWGvinKiBqm8WeXieA+lfPLKg/Tc7cDrTnOP216jAYtRZXq4YwRBbAvW823T9xAc2OWio5+DCs3rxovhYXEpeQJaBksXUXVNWT2Qn1C18gvK2d4s8ZLd/85EVXFJRw8F1/Lcx2tLHtpBIbH3vWGP916uhht8QyyupT4MjcPBvax2TFzW2ch26IwwMqoYM3twCgNA3NomMj+MkxnAM9v5PhQaDefXrEFNRtBnF64tjgnemrK6L3Tyd0t+7jfnQJcHM9Kvn7kZsJv6bie24PxDuNiIfubs/eMhMjXROi+T0NLCUoOOQgeUVESSaxcISwHZSXkywMkq50MPZLjzhXt/Fn1cwSVd++Cz0iDHtPJzw6tZtXfjSN7BzFnEQ7kSlzRkJJSDgADU/+PCyGOAjXAw8AdU4d9C3iFuawwQqD4fMRXl1Lx4S5+pWYHFWohWuvBXDnPvbiJsj0W1tAI1lU6czqFG+dUjAlN6HiknyxpRuhnM7cDoOMgR/oR5kijeqKXklgJL/xsE6+ta+KGzd/AJ/SzyxwVqoPPlL3Kxnu62HZdMwB5S6U3EUIIiVfP4dOz+LUsv1K2nSYtcXY57wxjpuDUvlpK+wUBEb6svioaOMWRuZA2Iwy3Qk0kRkg95wc2mz2L03mGVTRwkkPhuSn5xbHSGRgeRX5lFV8PfghLO2fYOicMGpMG2sQ4IpOFWOLiMX7SGdTxBBnzwua5kPW06pUxfjZxG698spnPL3uRe93jFyzBOYXOl1c/xY6GFXy3cQuOTieh4xc+t3SZwmRbHk9JispgnN+o3MlKVz9tegbneQm1s9LgtYyfvz59N7XPleLrHsWUcsHaIhRCTzA8hre/jL+NNrPe3cVDgX08seEOyq1ViDcPnp2VUvx+lEiIga0ecs1pylSNnVkvO5Mr+MHjtxM+ZkJfd6EuvIPp1tPFaIvvxJIKFvKCl7PpsBBtUebyOCckiQbJ2uZeJirq8fr9Mwt/MA2U7gEqsnmefaiVZa5RoutNhHRR+vrCjxmzQegOWN9C/60BPvPoz7nbdwRVuBg1k7ya3EDoKS+hY5OFGZp3/N0Wsr9RPB6UcIjjn6umbv0AX6jeh4ng9P1lvNzTTGJgHUpOwXJY+KoSVAbG2BgY5rbgMZbrI3jEuQj2P0yU8kZ8BbcE2unPh/k/++8gvMuB7B3ESs6PD/WMfKSEEMuAjcBOoGLKyEJKOSCEuGjAFSHEZ4DPALiYQeRqoaCEgqTKFb5Q+zobnP04ReGhTpgefJ3g7c/OuXN5WiaJM0GQCDmyZ+85lWF7zjSa0ShKLkfJoVKGPEEOrg2wUo9RpRYeiVNorNZhtd7PJwP9QOFN+Wg+j4okopj4Fe2sfxXn3ddC0mukeTvbgK9LwTtwztC8lD6ncCMv0QHN+hlOA0uDsDOFS+SBi6dzmCmX04i8eJ2fM42WiZVKob+w55JqrjTfJg0Dkc1hXGbn2HzXU3mqi8jwOEdvW8brkRZud227wJBSENznibHJ+Qb1W8b414rrGHBeOMFgVaf55LqdbPGeplUfpVZzoqEC594cs9Jg1Mrx4+hG+jpKadvdhxWdWBCNFwqWWIkkrqjJK2MtNFUPs8kRJ1EvcU64KTnoLWQF0B3IihIyFV7iy00aKsZRUDiSqeXF4ZWU7TPwHR7GiMevOKAXW1ucD+atLVomzkkThMK6YB/PBRvxBwOFgXIO84ia0RhKMsXExCoGciF8lQnSZaFpaZyXejpdpiYilICf8RYfk80Gnw+fRBUuTGlxPO/mrYkGwgcnEANjmLOsq3OlUTh0pNdNuHWcLy9/go0OA12oEDrFE4F2ttW1EDdc+LUMdwcPs1wfo0U/M/Zd2E/uSCznxa4WRit9DKX9uA54CHbkZxWYdrpM25ASQviAHwJfkFJOimnmI5JSfh34OkBARKb9qqB4PYzcVcf4RpP7PMPo4tz2aIcwMbwC06XOqbe8IQ0O8CYr2YAmdKY7MTJbjVYySeCnB/D2tfDr4tM8fMcu/rxy9yWPVxCs0vULfn8nCZllyLS478UvENznoPqZARibwFwEfbPFnOEb8PlcKxqvhoXQeCbVRs1zTTzTfwP/4VPb3rXEp6FSobr594FTPNrWzkTrhd+7hCSoqOio6MJ90fr6WsbPj6MbOfIna1l1agyjfxAsc1Geo8zn8PTEOfFSE0+/P86Dta/xnx96gh9s3kzUaCNVrpBotNi6tZ17Ike4yX0agN05D1/dfyf+7R6q93Vj9A1c0Yi6VurpTGejzmc+NVqxSXxvdOBY18xtvmN8d80taNk6gk9Pzu2slGViZS2suE5vJsz9DUd5fGjr2a+L8TkKTUO43Yx/cA3RVvj4Qy9zo/cEFhJD5hk3s/z7l75A8ICDqpP7MNOXz0O3IBp1B9LjJJOHnnwJ6x0DnDGQ3u8Z4i53/9lDXUJFF5ce+Z/au4GGHwlGJhvQk3nqO48i0+l59Tqclh0ihNApGFHfkVL+aOrjISFE1dRsVBUwPNeFs3RAt9CFesbyBUAXJnkfmC5lzgwpS1oc4E0qqadc1ADgwElWpnEKN1bhMcy9xlQKfWCC0JEqXlu5grfCb7HGkT9vpulCLtexZaXBC6kKvjN4Pf5DDiJHsjA2gZVKXVFfVqYRV9FpzgXmVUawno5GKN6k7mexLGQuz0gqQGe+FMMFhktFZYHrqWXi60piOnz8j74Hub/kIP/Of+GlFQRu4cAtoGQaERoMTPqNLAdz5exIrOCnXatJ9ARoPTUKg4VkqYvVFgGUeBpfd4jORISszLPZ1UW8wsXf3FKLEkrTVDnKo2V7uMHVR5XqZldW8Bc996Mf9xA+kUPGJq8Yh+daaItnSMscI6aBGlfRYonLxig6n/lui9I0kfE47mHJN4duhZo0Q6oL93ALzr4Y5omOOYmHpFaUY1WXoUcylDnivB2tQ5tQp6VxPuupcDpRwyFwOkBRMHsHEKqCaKonV+EjWeVg5HqL0mXj3O07TIOWBtzszOpsS6zBf9RBuD1fcIu5jNG5YBrzOUQyQ7K7nG+Hb6Bt2RNUqxmCigun0M8G1IRCJpOTeYPDuUpaHUNElEI16jXcvJRow93pwNM1hogVEqOb0fn39Z/Orj0B/CNwVEr5V+d99WPgE8BXpv59aq4LJ0zAFJhSns21m5cmLiVHpsIgG1SZi5SmUkqOsBsvfhpEy9nPy6hmgC6W0UqeHMyDRgCzp5/yJyfpqGnlzwP3878anmCZNrNceQYm41aOr7Tfj/4vEere6Mbo6cVkevoG6EKbo6W1xWC6GoGJxSrjdLFyeZR4nMHBel4sbSMXkmRDKo5FqKdy9yHC7QHavat588bl/Lv7v35V14tZOZ5PtfDVI3fieClIzavjcPog5tQS/WK3RWtkjLIdOu33lDDebLDK4WSt4zS//cjfv+PIQoy2H0W30PedRpbtmsDad+SK28evtbbYb5o8n1yFZ0DAqR6sK8xewAK1RSmxMhlK9yY46mrlg594k4/euJMP+z5LaH85lT39BSPhKo2p7Oo6et/n4OGWHdzgO8n3X7yJkiOLX0/VcIj0ujqyIQ3DJSj5WQrhdNDzQCnJDWk+u/E5HvIfpFE7s4xeWIr7m767efvtFbQ+NYB5suOy91hIjeZkApHOUPtSKZ39jTz7K6u51dPO5ncM8BYWMSvHv03cyPeObObRVfu4zluYGf7ByBb2P9tKzbYMVnsHcg7DG1yJ6Uzo3Az8CnBQCLFv6rM/pGBAfV8I8WtAN/CLc1kwmclSuidKNhLhzfe5cYk8eanxpVMfoquzjIanJZ6OuYliHmOMQbrxEWSHfB6AFayhgZUcZAd9shOTPBQ0zznSyGMlktS8lqFrcAV3b/4dPKUpNlf3cF/kMB/1jZy3o+0cj02W8vOxtezY24I2qaDmBIFTktC+EayJ2Iz0uXDj5N27HuabYEeWg68288YHT7LZUWgQb6SaOfpaE/Wd049ZM12NTG2cKGosE2kIyKhM5NzkKvOkhx1Yi1RPrXSGsrcmcI8FWDX0WWhKsqZ6gA9VvE2zY/CSO7sO53N8a+wmhrIBJnJu+mJBYhMe/PtchPos/KdiiIFhzPM2iyx6W8xkEYOjlP2khLs7f598qYEjmOWepuM0uEdpcozQnw/TnY3wo8MbcLa7adg1geib3s7hYm6LZxhfF2JsHQSVLK+nlvMXrz1A/QmjEKF9GktmC9kWte5hqiQ8WXMDz7W0cvf6IxxvKOdkw3oiByF8LIFyug8rkZx27CetphqrLMTI5iDjGy3uuW4fJ+LlvNjbQtUbEt/pScYXuZ6i62SDKoM3QWXrEEdvawBVsrX5OFtDndzlPUqZIshLk0N5wVvpJn7Qu5nh16ppeCsHo+NXvMWCtkXLROZBS5q4h1X+4Sf38v/qb+PT67YxnPMzkvPx9kAdyZgLR68D97Cgss/kueqbeNp7EwCucUnt0RSOzpF37UCcb6aza28bXHKe+X1zW5zz7pvPIQ8cI7j8el5LtOIUBlmpMbijispjEtezuzHnKFhfSJRyNx++6HdndifslC8yKcevXPtmg5TIbBZt+yEq3nbh628lURPg9Y2tpFY7uN395Fl3ujOGowo8OXwPew810vJYCq17uBBaIZ3BTCZnrA8KGhcaR0+Uyp1l7LiziQ/7DxXKEW2kcqeJoyc67bn/6Wp8QT5+TcSck6aJyCmkDAeecJqc30HFItVTmc8h9x/Fd9xF4M0gI/c1sX/9cpStkq2hLiqDe9+VKw9gR7qFn7SvJR9zok2qePoElYMWwafexsrlkZb5rhehxW6LMp/DjOYI//w4kTf9pFvKSVR7eeau1ZSWxFldMkjHZAkD0QCVP3Hi74hj7Zv+DrtibotniDcIAivH8CsW+5N1lO1Q8XREsab5hr+QbdEYHILBIapqrmNsIswjv/oE3tIc26pa+KbndsBHSaoMdUzHmoghTQukhZzKOSqUMzkvFVAUhKZh1JUSb/AwfmuO21vb+R9VL3D/3l8lcShC1a5uzJHRRa+naCqGW6GqdYh/bXuMiVYNHYsW3YWFxEIhIy3GrRw7U6v58eA6Bt+spu71NMqre6c1AbHgGi0TLWXgHlcJPmsQXenih6ENTEx6yE86CO/TiAyY+HecxoonsJJJfF4vQi+YMTKXx0qlFsV3o+gjm/tfaWfHpzed/X35UC8ymZozI6qYkPkcpmni3X4Cn9NBxUteEr4KPhn8jxc9XotlWZUYRfYPYWazhc7hKv2MFhqrpx/v2AQTn6s+q1OLZfF2nji73POeYypvlPQbVLgnOXyiiUjP4vu/W9kscnSM0mcsyl73kghW8JKzmhectyAvsvlEyZqsmEiBEUcYJmSyyHwes4iSv14KMzaJSCRxjU/gcrsofTOAdGoM6PW4DYvl+QwM9SIz2TnL11UUCEGuLcWfrHwOv6JyNFZJ6c9OFXy/ihjfqyfwHwjypYFPkWgQ+LeO0NLWS2hjYSlyKOWn60gjvm6FyLE8Sr7w1NKlGpmIQqJeojQkaa4YYWt4Dx4lx7jhZdtQEzfs/l3qn7GoOtKHOTQy58EcZ0U6g7c/x2jGiQ40aBJ9ajhvz+fYnl7ONztuYqgvTOVLKt7BHMtP92CNTyxiqNcrox48jU/TkIZB5TEP1vYwpWYKzARiMoHM5jDj8UK4Ego+xpxZrVnEsa/oDSkzGoVd55zFlp759A4s85xz3GDhn0ttgj8TwPFaRmazhYE1Gj2rcynomgv0QZ3X/CvwdQs8Q0WQ40zKQqDCkREYObeUdTkf82v2OVpmYcYsn4PJSRg650+7pAyn8xEC4XAQ9KdY7+xDRyOd11GGeha7ZFfEjEYRiSSle1w4J4MMBEqYqPJQXRLjprLT1LmjxBpdRF0BTKeOMtWc0hUS02/gLU/SVj7Iav8AES3JuOHlzeFGBk6WEdmn4D02hNHZvbgiz0NmcziiGZIdQf5Hzd141SyKkKhYtCfKOTVeSvJomGCfIPz2MIxFMRY6b+UsOD9EgRWPw+DQ5U+Qck7DXcyWojekbGzec1gmMmvS9N/eRqgqMm+cfQOzsZkvFI8HJRKmLhBjlcODKS0Uce2YjTKfQx48jveQQvNPtUJqoqCfJz5xC5mGHJubO1nbcJBb7j7OmOkD4C53P8fzbr45fBs7+xt468Bygkc1fAMmgVdPszIzikynMYqs/ZnRKLw9QfMhBx2aBorv3JdWmkrZjTQ7wJKYFwm2aTO32IaUjU2RIrNLbNnIprhRzs0t5qXJ0Xye8aSHmkUs0oyZmqGQWRPLNBG5HBW7K0h16Rw90cJBXzPf8t+KMAQIsHwmIqPgGlZxjkFFzMLfm0aLpgs+VcVshEz51l6rCZSXErYhZWNjY2MDVsHHxJAKCSvLG6lmkrHF2z14tUjDQBoGzqffwgnMJB9NMfsR2RQftiFlY2NjY1PIDTgySubP1vBA2e+ipyTNg9MPQWJj817FNqRsbGxsbApR5TMmjmd3M7NwwDY2723EpZJjzsvNhBgBksDogt109pRyYTkbpJRlVzppqWu8xvTB0tdo19NLsNQ1XuP6YOlrtOvpFEtd44IaUgBCiN1Syi0LetNZcDXlXOoarxV9sPQ12vV0/s5dSOx6Oj/nLiS2xvk7dyGZTTmnkWbUxsbGxsbGxsbmYtiGlI2NjY2NjY3NLFkMQ+rq0scvHFdTzqWu8VrRB0tfo11P5+/chcSup/Nz7kJia5y/cxeSGZdzwX2kbGxsbGxsbGyWCvbSno2NjY2NjY3NLLENKRsbGxsbGxubWbJghpQQ4n4hxHEhxEkhxBcX6r5XQghRJ4R4WQhxVAhxWAjx+anPvySE6BNC7Jv6ef80rmVrXCTmSmOx6oOlr9Gup7bGd1ynKPXB0tdo19OZaURKOe8/gAqcApoAB7AfaFuIe0+jbFXApqn/+4F2oA34EvCfbI3vHY3FrO+9oNGup7bGa0Hfe0GjXU+nr1FKuWAzUtcBJ6WUp6WUOeC7wMMLdO/LIqUckFK+PfX/OHAUZpXw3Na4iMyRxqLVB0tfo11PZ8RS11i0+mDpa7Tr6cxYKEOqBug57/deZlng+UQIsQzYCOyc+uhzQogDQohvCiGulDzc1lgkXIXGa0IfLH2Ndj19z2u8JvTB0tdo19MralwwQ0pc5LOiirsghPABPwS+IKWcBP4eWA5sAAaAv7zSJS7yma1xgblKjUWvD5a+Rrue2hq5BvTB0tdo19NpaVwwQ6oXqDvv91qgf4HufUWEEDqFP+R3pJQ/ApBSDkkpTSmlBfwDhSnKy2FrXGTmQGNR64Olr9Gup7bGKYpaHyx9jXY9nbbGBTOk3gKahRCNQggH8FHgxwt078sihBDAPwJHpZR/dd7nVecd9kHg0BUuZWtcROZIY9Hqg6Wv0a6nZ7E1FrE+WPoa7Xp6luloXJhde7LgFf9+Cl7xp4D/slD3nUa5bqEw1XgA2Df1837gX4CDU5//GKiyNS59jcWq772g0a6ntsZrQd97QaNdT2em0U4RY2NjY2NjY2MzS+zI5jY2NjY2NjY2s8Q2pGxsbGxsbGxsZoltSNnY2NjY2NjYzBLbkLKxsbGxsbGxmSW2IWVjY2NjY2NjM0tsQ8rGxsbGxsbGZpbYhpSNjY2NjY2NzSz5/zlHSon0X6T7AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 720x360 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(MNIST['Train']['Features'][0][130:180])\n",
    "print(MNIST['Train']['Labels'][0])\n",
    "features = MNIST['Train']['Features'].astype(np.float32) / 256.0\n",
    "labels = MNIST['Train']['Labels']\n",
    "fig = pylab.figure(figsize=(10,5))\n",
    "for i in range(10):\n",
    "    ax = fig.add_subplot(1,10,i+1)\n",
    "    pylab.imshow(features[i].reshape(28,28))\n",
    "pylab.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because perceptron is a binary classifier, we will limit our problem to recognizing only two digits. The function below will populate positive and negative sample arrays with two given digits (and will also show samples of those digits for clarity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def set_mnist_pos_neg(positive_label, negative_label):\n",
    "    positive_indices = [i for i, j in enumerate(MNIST['Train']['Labels']) \n",
    "                          if j == positive_label]\n",
    "    negative_indices = [i for i, j in enumerate(MNIST['Train']['Labels']) \n",
    "                          if j == negative_label]\n",
    "\n",
    "    positive_images = MNIST['Train']['Features'][positive_indices]\n",
    "    negative_images = MNIST['Train']['Features'][negative_indices]\n",
    "\n",
    "    fig = pylab.figure()\n",
    "    ax = fig.add_subplot(1, 2, 1)\n",
    "    pylab.imshow(positive_images[0].reshape(28,28), cmap='gray', interpolation='nearest')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    ax = fig.add_subplot(1, 2, 2)\n",
    "    pylab.imshow(negative_images[0].reshape(28,28), cmap='gray', interpolation='nearest')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    pylab.show()\n",
    "    \n",
    "    return positive_images, negative_images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by trying to classify between 0 and 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACqCAYAAACTZZUqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAI8UlEQVR4nO3dS6hV1R8H8H3kKuHzopZIiIgEooYRUYN0IGoZCkqiGSIiDq5Y6ERwEBENGog5UtTIBtee+IAok3yEICImSoMiqFEiKb3z+iqse/+TKP6steUc9z33dzz38xl+WXedlZ77ddPae69aX19fAcDAGxK9AIDBSgEDBFHAAEEUMEAQBQwQRAEDBOloZHCtVnPPGk3V19dXG+jP9L2m2cq+166AAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCNLQoZzce06cOJHN582bl2Rr1qzJjt23b1+/romBM3bs2CQbOXJkduwLL7xQ15xPPPFENt+1a1eS9fT0ZMcePXo0yfr6Bt/ZqK6AAYIoYIAgChggiAIGCGITro2cPHkyyZ588sns2N7e3iQbjJsg96JRo0Yl2TPPPJMd+8477yRZR0dzfu0nTpyYZJMmTcqO7e7uTrKtW7dmx3733XeV1tXKXAEDBFHAAEEUMEAQBQwQRAEDBKk1svNdq9Vsk7eAl156KZu//PLLSTZ06NDs2P379yfZunXrsmNv3rzZwOqq6evrqw3Yh/2jVb/XnZ2d2fztt99OskWLFjV5Nc33ww8/ZPMlS5Yk2TfffJMde/Xq1X5dU38p+167AgYIooABgihggCAKGCCITbgWt3Tp0iR7//33s2OHDRuWZF9++WV27Jw5c5Ls2rVrjS2uCWzC/WfhwoXZ/MiRIwO8ktazYcOGbL5nz54BXkl9bMIBtBgFDBBEAQMEUcAAQRQwQBAvZG8RZS+ufuWVV5Isd7dDURTFr7/+mmS5x5OLojXueOA/s2fPTrItW7YErOT/bdq0KckuX76cHbt58+YkKztBuapt27Zl819++SXJDhw40JQ19AdXwABBFDBAEAUMEEQBAwTxKHKAxx9/PMnefPPN7NiZM2fWPe+qVauS7IMPPqh/YS1gsD6KfPDgwSR79tlnK897/vz5JPv888/r/vk33ngjyb766qvs2BEjRiTZ2LFjs2NzG2O534tGHTp0KMmWL19eed6qPIoM0GIUMEAQBQwQRAEDBFHAAEE8itxEq1evzubd3d1JVnY3Su6U1xMnTmTHHj16tIHVEaFWy9/kMWRItWuh3B0wRVEUP/74Y5J99tlnlT6rzI0bN+rKiqIoPv300yR77LHHsmMb+bOZNm1aki1evDg79vDhw3XP2yyugAGCKGCAIAoYIIgCBgjiUeR+MmHChCQ7fvx4dmzu8eKyv4d9+/Yl2dq1axtc3b2j3R9FnjVrVjb/4osvKs07efLkbH7p0qVK8w6kZcuWZfOq7/Mte8y/q6ur0ryN8CgyQItRwABBFDBAEAUMEEQBAwTxKPJd6OzsTLJjx44l2YwZM+qes+yU4o8++qjuOWh9U6ZMqTxHT09Pkt2+fbvyvNHOnDmTzXP/vaNHj272cgaEK2CAIAoYIIgCBgiigAGC2IS7C7nTXxs5vThn0qRJ2bxsc4570++//155jnPnziXZb7/9VnneaFeuXMnmR44cSbKVK1fWPe/TTz+dzUeOHJlk169fr3ve/uAKGCCIAgYIooABgihggCDeB3wH48ePz+a5p94eeeSRuuc9e/Zsks2dOzc79s8//6x73nbQTu8Dzj2t9e2332bHPvDAA5U+qx3eB1xm0aJFSfbxxx9XnnfcuHFJ1qzNTO8DBmgxChggiAIGCKKAAYIoYIAgHkW+g507d2bz3Mm2ubtJyt5vOn/+/CQbbHc7DAYdHemvV9W7HQaj77//PnoJTeMKGCCIAgYIooABgihggCA24f6Re+x46tSpdf987lDErVu3ZsfacBsccu/+fffdd7NjV61a1eTV0IpcAQMEUcAAQRQwQBAFDBBEAQMEGXR3QZQ9Cvree+8l2aOPPpod+8cffyTZ+vXrk+zw4cMNro520tvbm2THjx/Pjq16F8SBAweyee6x94E++bdenZ2d2by7u7vSvHv27Mnm/XFCdVWugAGCKGCAIAoYIIgCBggy6E5F7urqyua7du2qe45Tp04lWdmpxjSmnU5FzhkzZkw2P3nyZJI1ctJ2mfPnzyfZli1b6l5Ds9x///1J9vrrr2fHrl69uu55b926lWTTp0/Pjr148WLd81blVGSAFqOAAYIoYIAgChggiAIGCNLWd0E8//zzSbZ79+7s2FGjRiVZ2anGK1asSLIrV640uDpy2v0uiDKzZ89OsrLv6owZMyp91unTp7P5xo0b6/r5np6ebD5s2LAku++++7Jjc48XP/zww3V9/p0cOnQoyZYvX1553qrcBQHQYhQwQBAFDBBEAQMEaYtNuLLHOy9cuJBkU6ZMqXveZcuWZfMPP/yw7jlozGDdhMvJbfYWRVG89dZbSTZixIhmL+dfP/30UzYfPnx4kg3kuoqiKFauXJlk+/fvH9A15NiEA2gxChggiAIGCKKAAYIoYIAgbXEq8pIlS7J5I3c85IwePbrSz0MVZbv3Dz74YJJt37692cv5V+5l6s109erVJCs7WOGTTz5p9nL6lStggCAKGCCIAgYIooABgrTFJtzt27ezeW9vb5INGZL/N+fvv/9OsoceeqjawqAJ9u7dm2QLFizIjl24cGGzl9Nvbty4kc2fe+65JDt27FizlzMgXAEDBFHAAEEUMEAQBQwQRAEDBGmLF7KX+frrr5OsoyN/48drr72WZLmTW2kuL2S/O2WnD8+fPz/JnnrqqezYF198MclqtfSvo6wzcmN37NiRHfvqq68m2V9//ZUdm3sU+V7jhewALUYBAwRRwABBFDBAkLbehOPeYxOOdmQTDqDFKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCdDQ4/ueiKC42YyFQFMXkoM/1vaaZSr/XDR1LD0D/8b8gAIIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYL8D+KUFeSspSmeAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos1,neg1 = set_mnist_pos_neg(1,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def plotit2(snapshots_mn,step):\n",
    "    fig = pylab.figure(figsize=(10,4))\n",
    "    ax = fig.add_subplot(1, 2, 1)\n",
    "    pylab.imshow(snapshots_mn[step][0].reshape(28, 28), interpolation='nearest')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    pylab.colorbar()\n",
    "    ax = fig.add_subplot(1, 2, 2)\n",
    "    ax.set_ylim([0,1])\n",
    "    pylab.plot(np.arange(len(snapshots_mn[:,1])), snapshots_mn[:,1])\n",
    "    pylab.plot(step, snapshots_mn[step,1], \"bo\")\n",
    "    pylab.show()\n",
    "def pl3(step): plotit2(snapshots_mn,step)\n",
    "def pl4(step): plotit2(snapshots_mn2,step)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afbc754ef0d04c95a1af039574b46a6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='step', max=66), Output()), _dom_classes=('widget-interac…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.pl3(step)>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snapshots_mn = train_graph(pos1,neg1,1000)    \n",
    "interact(pl3, step=widgets.IntSlider(value=0, min=0, max=len(snapshots_mn) - 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note how accuracy goes up to almost 100% very fast.\n",
    "\n",
    "Please, move the slider to some position towards the end of the training, and observe the weight matrix plotted on the left. This matrix will allow you to understand how perceptron actually works. You can see the high weight values in the middle of the field, which correspond to pixels that are typically present for digit 1, and low negative values by the sides, where parts of 0 digit are. So, if the digit presented to the perceptron is in fact 1, middle part of it will be mupliplied by high values, producing positive result. On the contrary, when perceptron observes 0, corresponding pixels will be multiplied by negative numbers.\n",
    "\n",
    "> You may notice that if we give our perceptron a digit 1 slightly shifted horizontally, so that its pixels occupy the place where there are vertical parts of 0, we may receive incorrect result. Since the nature of our MNIST dataset is such that all digits are centered and positioned properly, and perceptron relies on this to distinguish between digits.\n",
    "\n",
    "Now let's try different digits: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACqCAYAAACTZZUqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAI90lEQVR4nO3dT4hVZR8H8HPfphJxLKzQippFZIOGTFEZFNEuxE3YIjAKFxYoKkqMItGqiCjEhSARWRkVgqCbyAKFslqJC/9ubKOgKP6JcuHoQPfdvBAvz+/Uvc2993e98/ksvzxzzsPM8cvB8zznNJrNZgVA7/0newIA05UCBkiigAGSKGCAJAoYIIkCBkgy1M7gRqNhzRpd1Ww2G70+p+uabqu7rt0BAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyQZyp5ANz3yyCNF9sQTT/Ts/CtXrgzz559/vsi2bt0ajj18+HCR7d+/Pxx74cKF1icHLRgdHS2ytWvXhmNvv/32Ips7d244dunSpS3P4dChQ0W2Z8+ecOy+ffuK7OjRoy2fq9fcAQMkUcAASRQwQBIFDJBEAQMkaTSbzdYHNxqtD+6huiet0WqBBQsWdHs6Xffdd9+FeTtPlvtVs9ls9Pqc/Xpdd8vw8HCYv/fee0X22muvFdmsWbNaPlejEf852+mddkxMTBTZ7t27w7ErVqzoyhwidde1O2CAJAoYIIkCBkiigAGSDMRW5E8++STMB+GBW2TRokXZU+AmMTIyUmQ//vhjOPaBBx5o6ZjffvttmE9OThZZrx/CPfbYY0X28ssvh2N///33IhsfHw/H3rhxY2oTq+EOGCCJAgZIooABkihggCQKGCDJQKyCgOkuehl6VVXV119/XWQPPvhgODZambBr164ie/XVV8Of//PPP/9uij0RbZNevnx5OHbZsmVFNnPmzHCsVRAAA0YBAyRRwABJFDBAkoF4H/Drr78e5h999FGPZ9Ib586dC/NWt5L2M+8D/nfqrvXo30bd9uAvv/yyyNavX19kV65caW9yeB8wQL9RwABJFDBAEgUMkEQBAyQZiFUQs2fPDvOjR48WWTsrBU6ePBnmn376aZH99ttvRbZjx46Wz9UOqyA6q1+v63ZcvHgxzO+6664i+/zzz8OxGzZsKLLopeW0zyoIgD6jgAGSKGCAJAoYIMlAvA/4jz/+CPMXX3yxyOq+oPzVV18V2c6dO8Ox0VbMuuNCpy1ZsqTI7rjjjnBs9JA9ethWVVN/4HbnnXcW2dBQXDHRvC5fvjyl89+M3AEDJFHAAEkUMEASBQyQRAEDJBmIrcjtqHtaPNUnwNH24Llz507pmHX2798f5i+88EJXztdLtiL/pe5Lxz/88EORPfXUUy0f95Zbbml57L333ltkq1atCsdGebQVuqqq6vr160X28ccfh2PHx8eLrFtfKe4WW5EB+owCBkiigAGSKGCAJAOxFbkd7TxsGxkZCfO33367yIaHh//1nP7OTz/9VGQrVqzoyrnoL3XXVDsP3L755psiW7lyZTh206ZNRXbPPfe0PK923HbbbUW2Zs2acOylS5eK7J133pnyHPqBO2CAJAoYIIkCBkiigAGSTLudcO04cuRImD/66KMdP9fExESYR+9YnZyc7Pj5+4WdcH+59dZbwzzaCfnss8+2fNxGI/4Vt9oFhw4dCvNjx461PIeXXnqpyOp2qZ4/f77IHn/88XDshQsXWp5DL9kJB9BnFDBAEgUMkEQBAyRRwABJpt1W5DrRE+dou2QnRCsetmzZEo4d5BUP/L26v/1bb71VZAcOHAjHRtf11atXw7HRl8Hff//9Ijtz5kz48+2IVm1EK36qKn4n8UMPPRSO7ddVEHXcAQMkUcAASRQwQBIFDJDEQ7j/id6FOn/+/K6ca86cOUUWfaQQIj///HORLVy4MBwbfYDz2rVr4dhOPFxrVbTtuW4r9OXLl4vs7NmzHZ9TBnfAAEkUMEASBQyQRAEDJFHAAEmm3SqI0dHRMO/Gl4brvsDczkvwoRW//vpr9hRCdf/e7r///paPcfjw4SI7ffr0v55TP3EHDJBEAQMkUcAASRQwQJKBfgg3NjZWZHv27AnHjoyMdPz8b775ZpjfuHGj4+eCfrRz584wnzVrVsvH2Lt3b6em03fcAQMkUcAASRQwQBIFDJBEAQMkGehVEM8880yRdWO1Q1VV1YkTJ4rs4MGDXTlX9KXbqqqqhx9+uCvnO3/+fJFduXKlK+fi5rVhw4Yie/LJJ8Ox0Xb8HTt2hGM/++yzqU2sj7kDBkiigAGSKGCAJAoYIMlAP4TrpeirtFu3bg3HHjlyZErnqtvGuW7duikdt873339fZMuWLQvHTkxMdGUO9I/nnnsuzLds2VJkjUYjHHv16tUie/fdd8Oxk5OTbczu5uIOGCCJAgZIooABkihggCQKGCDJQK+CiLbQRk9fq6qqhoeHO37+pUuXtpVna+d3MzQ00JfOtDRz5swiW7NmTZGNj4+HPx9tL65bwbBx48YiO3PmzD9NceC4AwZIooABkihggCQKGCBJI/qP89rBjUbrg/vUL7/8EuZPP/10j2fSG9GDyKqKt33Wje3lV2mbzWa8d7WLbrbrevHixUV23333hWOjv90bb7wRjl27dm2RLViwoM3Z/b8PPvggzDdv3jyl495s6q5rd8AASRQwQBIFDJBEAQMkUcAASabdftJXXnklzLdt21ZkdS+ernsheq9cv349zA8cOFBkdU+bjx8/3tE50Tvz5s0rsi+++CIce+3atSK7++67w7Gtrog6depUmEdfNf7www9bOuZ05Q4YIIkCBkiigAGSKGCAJNNuK3I76t7bO3v27CJbvXp1kW3fvr3jc6qq+i8P93LLcLfYivzPxsbGiqxui/2MGTOKrO5LxdHXuqNrKnrYVlVVdfbs2TDHVmSAvqOAAZIoYIAkChggiQIGSGIVBH3FKggGkVUQAH1GAQMkUcAASRQwQBIFDJBEAQMkUcAASRQwQBIFDJBEAQMkUcAASRQwQBIFDJBEAQMkUcAASYbaHH+pqqrT3ZgIVFU1knRe1zXdVHtdt/VCdgA6x39BACRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyT5L95QHHzClR5NAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos2,neg2 = set_mnist_pos_neg(2,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bf6f25d14c3b4d548ac026af97f70e2a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='step', max=66), Output()), _dom_classes=('widget-interac…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.pl4(step)>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snapshots_mn2 = train_graph(pos2,neg2,1000)\n",
    "interact(pl4, step=widgets.IntSlider(value=0, min=0, max=len(snapshots_mn2) - 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Discussion\n",
    "\n",
    "For some reason, 2 and 5 are not as easily separable. Even though we get relatively high accuracy (above 85%), we can clearly see how perceptron stops learning at some point.\n",
    "\n",
    "To understand why this happens, we can try to use [Principal Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) (PCA). It is a machine learning technique used to lower the dimensionality of the input dataset, in such a way as to obtain the best separability between classes. \n",
    "\n",
    "In our case, an input image has 784 pixels (input features), and we want to use PCA to reduce the number of parameter to just 2, so that we can plot them on the graph. Those two parameters would be a linear combination of original features, and we can view this procedure as \"rotating\" our original 784-dimensional space and observing it's projection to our 2D-space, until we get the best view that separates the classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "def pca_analysis(positive_label, negative_label):\n",
    "    positive_images, negative_images = set_mnist_pos_neg(positive_label, negative_label)\n",
    "    M = np.append(positive_images, negative_images, 0)\n",
    "\n",
    "    mypca = PCA(n_components=2)\n",
    "    mypca.fit(M)\n",
    "    \n",
    "    pos_points = mypca.transform(positive_images[:200])\n",
    "    neg_points = mypca.transform(negative_images[:200])\n",
    "\n",
    "    pylab.plot(pos_points[:,0], pos_points[:,1], 'bo')\n",
    "    pylab.plot(neg_points[:,0], neg_points[:,1], 'ro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACqCAYAAACTZZUqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAI8UlEQVR4nO3dS6hV1R8H8H3kKuHzopZIiIgEooYRUYN0IGoZCkqiGSIiDq5Y6ERwEBENGog5UtTIBtee+IAok3yEICImSoMiqFEiKb3z+iqse/+TKP6steUc9z33dzz38xl+WXedlZ77ddPae69aX19fAcDAGxK9AIDBSgEDBFHAAEEUMEAQBQwQRAEDBOloZHCtVnPPGk3V19dXG+jP9L2m2cq+166AAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCNLQoZzce06cOJHN582bl2Rr1qzJjt23b1+/romBM3bs2CQbOXJkduwLL7xQ15xPPPFENt+1a1eS9fT0ZMcePXo0yfr6Bt/ZqK6AAYIoYIAgChggiAIGCGITro2cPHkyyZ588sns2N7e3iQbjJsg96JRo0Yl2TPPPJMd+8477yRZR0dzfu0nTpyYZJMmTcqO7e7uTrKtW7dmx3733XeV1tXKXAEDBFHAAEEUMEAQBQwQRAEDBKk1svNdq9Vsk7eAl156KZu//PLLSTZ06NDs2P379yfZunXrsmNv3rzZwOqq6evrqw3Yh/2jVb/XnZ2d2fztt99OskWLFjV5Nc33ww8/ZPMlS5Yk2TfffJMde/Xq1X5dU38p+167AgYIooABgihggCAKGCCITbgWt3Tp0iR7//33s2OHDRuWZF9++WV27Jw5c5Ls2rVrjS2uCWzC/WfhwoXZ/MiRIwO8ktazYcOGbL5nz54BXkl9bMIBtBgFDBBEAQMEUcAAQRQwQBAvZG8RZS+ufuWVV5Isd7dDURTFr7/+mmS5x5OLojXueOA/s2fPTrItW7YErOT/bdq0KckuX76cHbt58+YkKztBuapt27Zl819++SXJDhw40JQ19AdXwABBFDBAEAUMEEQBAwTxKHKAxx9/PMnefPPN7NiZM2fWPe+qVauS7IMPPqh/YS1gsD6KfPDgwSR79tlnK897/vz5JPv888/r/vk33ngjyb766qvs2BEjRiTZ2LFjs2NzG2O534tGHTp0KMmWL19eed6qPIoM0GIUMEAQBQwQRAEDBFHAAEE8itxEq1evzubd3d1JVnY3Su6U1xMnTmTHHj16tIHVEaFWy9/kMWRItWuh3B0wRVEUP/74Y5J99tlnlT6rzI0bN+rKiqIoPv300yR77LHHsmMb+bOZNm1aki1evDg79vDhw3XP2yyugAGCKGCAIAoYIIgCBgjiUeR+MmHChCQ7fvx4dmzu8eKyv4d9+/Yl2dq1axtc3b2j3R9FnjVrVjb/4osvKs07efLkbH7p0qVK8w6kZcuWZfOq7/Mte8y/q6ur0ryN8CgyQItRwABBFDBAEAUMEEQBAwTxKPJd6OzsTLJjx44l2YwZM+qes+yU4o8++qjuOWh9U6ZMqTxHT09Pkt2+fbvyvNHOnDmTzXP/vaNHj272cgaEK2CAIAoYIIgCBgiigAGC2IS7C7nTXxs5vThn0qRJ2bxsc4570++//155jnPnziXZb7/9VnneaFeuXMnmR44cSbKVK1fWPe/TTz+dzUeOHJlk169fr3ve/uAKGCCIAgYIooABgihggCDeB3wH48ePz+a5p94eeeSRuuc9e/Zsks2dOzc79s8//6x73nbQTu8Dzj2t9e2332bHPvDAA5U+qx3eB1xm0aJFSfbxxx9XnnfcuHFJ1qzNTO8DBmgxChggiAIGCKKAAYIoYIAgHkW+g507d2bz3Mm2ubtJyt5vOn/+/CQbbHc7DAYdHemvV9W7HQaj77//PnoJTeMKGCCIAgYIooABgihggCA24f6Re+x46tSpdf987lDErVu3ZsfacBsccu/+fffdd7NjV61a1eTV0IpcAQMEUcAAQRQwQBAFDBBEAQMEGXR3QZQ9Cvree+8l2aOPPpod+8cffyTZ+vXrk+zw4cMNro520tvbm2THjx/Pjq16F8SBAweyee6x94E++bdenZ2d2by7u7vSvHv27Mnm/XFCdVWugAGCKGCAIAoYIIgCBggy6E5F7urqyua7du2qe45Tp04lWdmpxjSmnU5FzhkzZkw2P3nyZJI1ctJ2mfPnzyfZli1b6l5Ds9x///1J9vrrr2fHrl69uu55b926lWTTp0/Pjr148WLd81blVGSAFqOAAYIoYIAgChggiAIGCNLWd0E8//zzSbZ79+7s2FGjRiVZ2anGK1asSLIrV640uDpy2v0uiDKzZ89OsrLv6owZMyp91unTp7P5xo0b6/r5np6ebD5s2LAku++++7Jjc48XP/zww3V9/p0cOnQoyZYvX1553qrcBQHQYhQwQBAFDBBEAQMEaYtNuLLHOy9cuJBkU6ZMqXveZcuWZfMPP/yw7jlozGDdhMvJbfYWRVG89dZbSTZixIhmL+dfP/30UzYfPnx4kg3kuoqiKFauXJlk+/fvH9A15NiEA2gxChggiAIGCKKAAYIoYIAgbXEq8pIlS7J5I3c85IwePbrSz0MVZbv3Dz74YJJt37692cv5V+5l6s109erVJCs7WOGTTz5p9nL6lStggCAKGCCIAgYIooABgrTFJtzt27ezeW9vb5INGZL/N+fvv/9OsoceeqjawqAJ9u7dm2QLFizIjl24cGGzl9Nvbty4kc2fe+65JDt27FizlzMgXAEDBFHAAEEUMEAQBQwQRAEDBGmLF7KX+frrr5OsoyN/48drr72WZLmTW2kuL2S/O2WnD8+fPz/JnnrqqezYF198MclqtfSvo6wzcmN37NiRHfvqq68m2V9//ZUdm3sU+V7jhewALUYBAwRRwABBFDBAkLbehOPeYxOOdmQTDqDFKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCKGCAIAoYIIgCBgiigAGCdDQ4/ueiKC42YyFQFMXkoM/1vaaZSr/XDR1LD0D/8b8gAIIoYIAgChggiAIGCKKAAYIoYIAgChggiAIGCKKAAYL8D+KUFeSspSmeAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD4CAYAAAAEhuazAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAow0lEQVR4nO2dX4wkx33fv7XDXcZ7ywS8ISVcJO0sGQtBqJdE3AgiEutFQSTeixxYBmQM6QMt4KBlAjAB9MDDvujlHqwgAajQVHiJjrpwFpadf7AswKAlJkEAJzC9DPhPkM8k5dszLULk3dkJqZV11F3lobu1vb1V1VXdVd1V3d8P0NjZmpnu6p6qb/3qV7+qElJKEEIIGRdLfWeAEEJI91D8CSFkhFD8CSFkhFD8CSFkhFD8CSFkhNzSdwZsueOOO+TGxkbf2SCEkKR4/vnnr0gp76ymJyP+Gxsb2N3d7TsbhBCSFEKIPVU63T6EEDJCKP6EEDJCKP6EEDJCKP6EEDJCKP6EEDJCKP49srMDbGwAS0vZ352dvnNECBkLyYR6Do2dHeD0aWB/P/t/by/7HwDm8/7yRQgZB7T8e2J7+0D4C/b3s3RCCAkNxb8nLl92SyeEEJ9Q/Htifd0tnRBCfELxD0TdYO7Zs8Dq6uG01dUsnRBCQkPxD0AxmLu3B0h5MJhbbgDmc+DcOWA2A4TI/p47x8FeQkg3iFT28N3c3JSpLOy2sZEJfpXZDLh0qevcEELGjBDieSnlZjWdln8AOJhLCIkdin8AOJhLCIkdin8AOJhLCIkdin8AOJhLCIkdLu8QiPmcYk8IiRda/oQQMkIo/j1hmgTG1T4JIaGh26cHTCt6AlztkxASHk7y6gHTJDCAE8QIIf7gJK+eKbtyVOIOZJPAOEGMENIFdPt0QNXNo6OYBKZqHDhBjBDiE4p/B6g2bqlSngRWbSg4QYwQ4hu6fTrA5LIpTwIDgEceOSz80ykniBFC/GMt/kKI80KIt4QQr5TSjgshvi2EeDX/e3vpvTNCiNeEEBeFEJ8qpd8rhHg5f+8rQgjh73bi5Phxdfp0Cty8eTCQ+9BDwNWrhz/zzjtBs0YIGSkulv/XAXy6kvYogGellB8G8Gz+P4QQ9wD4HICP5N95Qggxyb/zVQCnAXw4P6rnTJK2sfnb28B77x1Nv36d+/qSnD4ngHDyyfCQUlofADYAvFL6/yKAE/nrEwAu5q/PADhT+twzAO7LP/PHpfRfAfCkzbXvvfdeGSuLhZSrq1JmW7dkx+pqli6llEIcfq84hDg4h+4z1c/1wWIh5WyW5WM2O7gv0iF1hSyWa7OwRAeAXanSc1Wi7lCI/19W3v+L/O/jAB4opX8NwGcBbAL4Tin9FwB8y+baMYv/dKoW7dkse382M79v+kz1c13Tp+aQEjaFqO9rs7BEiU78Qw34qvz40pCuPokQp4UQu0KI3bfffttb5nyys3PUT1+wt5e9b7PE89mzwPLy0XOsrPQb6aOKVNrfpyuqc/qcAGJ7bRaWpGgr/j8UQpwAgPzvW3n6GwA+VPrcBwH8IE//oCJdiZTynJRyU0q5eeedd7bMahjqynWxNEPdEs/zOfDUU9kgcMF0Cpw/32+kDyedRUKfOwTZXpuFJSnaiv83AZzKX58C8Dul9M8JIW4VQtyFbGD3OSnlmwDeEUJ8PI/y+dXSd5KkrlwXhs98nkX1FNE9KkGfz4ErVw76zFeu9B/iyV3JIqHPHYJsr83CkhYqX5DqAPCbAN4E8B4yC/7zAKbIonxezf8eL31+G8DryAaF7y+lbwJ4JX/vceTrC9Udsfr8Tb76WAZs20A3bkT0OZhqc22bwsIB4c6BjwHfPo9YxV9V3mMasPUB6yuxxlRYaEn0gk78uapnS3Z2slm5ukHf1VXO0CUEgHk5Wy5ZGwyu6tmQuk1XTp8+LPwrK9lALffuJaQCB4Sjggu7GTBtujKfqyPbrl8H1taywVpCSIn1dS5ZGxG0/A3UhS37NmQ4g54Mmj4jlsgRKP4G6sTdZ2Rb0cvY28tGwopeBhsAMhjm8/oJL6QzKP4G6sTdpyHDyZFkFNhMeCGdQPE3cPKkOv3nfz7769OQ0fUy9vboBiIdQ//jKKD4G/jt31anP/vsQX1wNWR09crkKircQL/2a6yHJDD0P44Gir8BXew+cNgdUwi6EMAtt2R/NzaAhx8+LPQPP6yvVyoXUpXr17M5BSQxUrKk6X8cD6qZXzEefczwrVu2oVjawbQWf3WZB9MM4PLkSNN56uCM3IhIbVarzeYTJCnQ8ZLOyfPww+b3hTgIWbadJK37XOHvL7uQmsJee2SkZkmPdXG2lHpnnqD4K3j4YeCrXzV/xueqGKp6VV7a2Sa9IDWtGTypzWodYyz+SC0mir+Cc+f079WJbx3V7ep19eqxx45u7rK8nKWbSE1rBk9qlvQYY/FHajFR/BXcuKF/77HHsvrQlC98wa5eFZu7lD/71FP1dTA1rRk8sVjSLm6NscXij9ViUg0ExHh0OeA7mZgHeW2WcVYd02n4vKc2vjgK+h6BZ6Ew0+f+yB0Arudvz9aWXsCLoIdyfZ5OpVxeNgt/l3Wtb60hkTFwcWvNwBtHnfhzPf8Kdevzq5Ye1y1TPplkPef19ayXP/TeM4mUpSV1hIIQ7ULLhsTOTubjv3x5cBWW6/lboFqfv8q77x51l+pcgzdvjsdtmgQjDOcDEH4gaAjPdWzjHKD4H0I16F/l6tWjUWAcZO0RW+EZaTgfgLCDzmN+rqmj8gXFeHTh87edqVsM3tb5/KfTA7ch/fABcPHVjt3vHaoAjv25JgDo869H57u3YWUli8P/0Y8Op6+uAqdOARcuHO5VcG9fD7jsCUu/dxj4XKOHPn8LbBZX03H9OvBXf3U0fX8/E/kRziEJj0t8dte+uVB+cF/n9XUe+jzTRdUdiPHoKtRzsZByacne/dPm4FpZLXFxOXQZzhfqWr7O6zN/Aw+THAJgnL8906lesGcz8/uqQzdpjG7RlrgKT1cDL6H84L7O6zt/HNCKGoq/A3Wr2rrM8F1dzSaN6TSK9aYlMT7AUMsi+zovl20eFTrxp88/p+wCXdI8lcKNWax9ZcO5c8ATT6jXygIYJdeaGOOzjx93S7fFl3+dfnoCDvgCOBqqrFrYrRoWPZ/XL/C2tGTWopEuJtg9Q5iEBPiL149lsTnSL6ruQIxHSLePzgU6mZi9CXXun2PH9J8zfY+9b4/0MSAZ0q3iy80Vo7uMBAGM89fTJlR5Zwd44AH1e8X3TWv/qHoZqjB10hCXuQApX5MQDYzzN6BzdUppt/S5zv1TnFcXjn7jBnvfweljrXa6VUgCUPxhntxlMwhbV9d1jUsx8DumTZM6p4/BzS52wxrKOAbpD5UvKMYjdKhn4QI1xffbfF/lQuU8mB6JdS5AG1igiANgnL8dpsXdmurBYnF4Ylh5wTfSAbaCnoqocjE14oBO/On2qWDyBkiHWPyiVy4E8OCDh/cI+PGP2+WRPX5HbOcCxBx7W/7RdasPDn3PWeIVin8Fm8Xd9vezlTptlo4HjkYStdETLp8ekFg38q7+6DpcxzHKFsott2R/aU2MB1V3IMaj6z18TZu41y3RYLP2T9OQb/b4AxLrwzUNRjV1T5kmqcTo6iKNAX3+9VR98zbHdGq/zo8PPeGyLAEoj/ZXH3AMQhhiIKquQem7wSPe0In/KN0+Kp+5zf69Kq5erd/6sUqbkG8uy+IZlY9OiOx1LLG3plhh1zWNisJft2tR364uEh5Vi+B6ALgE4GUALyBvZQAcB/BtAK/mf28vff4MgNcAXATwKZtr+LL8dQEdrhZ/00OIzK3kO/99G6fJEqurp0zIdfxp+Q8ehHT75OJ/RyXtywAezV8/CuDX89f3AHgRwK0A7gLwOoBJ3TV8ib+N+9THMZ2GW8c/hVD0ZEjFj+bjR7ct/LQmBoVO/EO6fT4D4EL++gKAXyylf0NK+RMp5Z/mPYCPBczHIbroza6sAI89pl8XqG0eYlzFOFlS8aP5+NFNBW8yyf725epi/HLn+BJ/CeD3hRDPCyFO52nvl1K+CQD53/fl6R8A8Gel776Rpx1BCHFaCLErhNh9++23vWRUV6cLN68Pbrstqzup6MqoGdM6PKaxg5/+NLP7+7AmVPHLDz6YRuhpyo2WqjvgegD4m/nf9yFz6XwCwF9WPvMX+d/fAPBAKf1rAH6p7hqhff6+/fqma7FHHYimrpGx+NFiLZB17qgY8qgi1udZAV2FegL4EoAvIhvMPZGnnQBwMX99BsCZ0uefAXBf3Xl9hnqq6rrPsYCyT38sutI7XVTEIfyYMd6DKZS1OCaTOPJaJoVgARlQ/AEcA3Bb6fX/AvBpAP8Shwd8v5y//ggOD/h+Hx0O+OrY2vIn/tOpvm7FWPcGQeiKmIiVlySpDkQnEiwQUvzvzsX8RQDfBbCdp08BPIss1PNZAMdL39lGFuVzEcD9NtcJLf4u5c8lLLRcXqkfAQldEbu08mK0EELmKdUQ1LFb/l0docTfxuVT6EpR5hcLKVdW7BuAoiwkUlbSJPTD7crKi9FC6NKlVq5wuiOGxlDKOH8rBRR/BW0MDtUyzabGQ8pkeolpEnrd/q5abt/XCTk/IJTVsliYF9eKSWBj7KVVoPgrcBnktRHoujpCyz8wIdft78rK82kh+Mqzrzy5CGWdZcZKY83oxV9V7myCDFRlrdxDLQyU2SwbNDbVtUR6icOnaSvsw8qrO4dPC6HJuVxC4VTdYZ/b2S0W+grJ7rI1oxZ/Xbk7dsxO+OsEvPy5rS1z3U6glzh8+vK/qQaLVlbai6QO1/vUXbvOqrHJd9NGjd3l1oxa/NvE8FfDi7kS7gDoQlBUrbxuYGg6rf9uE1zv0/T5tj2WpstSs7vcmlGLv4t7x1T2bc7F3mgChBaUJtPIfaNrbEz32aZHVPfdNhvSsLvcilGLv4/Zu0W57MryZ3kPTMgH3KTAtclr9X2Vm6bcy9Dda5seUd13bUPr2HX2zqjFX1XumvQGinoXevc79nQTQjX673pU3T7V87tGEdQVbpOF3bTg2Xy33Ejp8saus3dGLf5SuhlHpqOY5FXtUZsMKlc4xpUILhNFikKyvHw4bXnZXHCaxg83tbDb9IhcvstC3hmjF38VdXNJdEaTTfBDGzgZLBFchLcoIK7iWlcYmg5o9V2Y2L3tDIp/BRv/ve4ItUNXAY2iRLAR3tARO7r36/JmW5hCjI3oJspQ+INA8S/h2lvv2piiUZQIXYz+N/H5F91TXWhpdbEql2uvrJiXrW17P67nYlRELRT/Em2jf5aWwtX1ApbrBOhi9L+4juvMQVsLx5TPNuGZOnx1a2khWUPxL9E27v/YMZY7Io+O/Nta1F3gYuFMJuqGxbaiuAh30wEt26gq+kaPoBP/kBu4R8vx4+2+v7+f7XE9m2XbjPa15zXpkWLf2atXD9J+7ueAxaKffXCrmDZrr3LjRiade3vZPRX70NpuNu1yrSYbW5f3+C3y6yMvI2eU4m9iNgO2toC1Nf1n1tezun3pEnDzZhx1nXTM9nZmBZTZ38/SY8BWuKuU7+Hkycy68Xmts2eB1dXDaaurWboO1bP2kZeRM0rxv3ZNnS5EJuRPPAG8807WCFTLfl05JSNBZ2H2aXnu7AAbG8DSEvDuu8Dy8uH3l5eBlZX681y+nJ3rwoWsR1BmqSIZrhViPnfvNts+05UVVk4HRin+tj3PJ54Ann6a7p1RUxbUjY16l0hflmfZNSJl5o4SAphODwrvU08B588fFOjJRH2u9XW9tX377foKoXtWVVy7zbbPtNpQETOqgYAYj9ChnhywJUcwFZTYClHTtft19+BrOehQ651wwNcaMNrnMLroOIZXkp9hs1hZLAWmbRRN9R58LgftA9ton75nLkcIxb+G2Aw5EgEprbPRVnxtFr8KsRz01taBmE8m2f9d3O+IoPjXwLJEjpBSoQixImfdtnRlmjyrrS31d2waAFpr1lD8a0jJyCMdkZrA6Fw4bXfhsr2267PSuW8mk3b3Sw5B8a8hJSOPdEjqAmMjyr4sH9dnpfPbA653SQzoxH+UoZ4qmsw9IZ6xDRXskpRn8+3sAKdO1U9G8xW2avusit9Zhy4EtY4Yy0/MqFqEGI9Q6/lXx7hSNvKSJjUXS+zUhUeWrfoun73Ldo62FVK1uxLLz88A3T6HodZEho3fbQgumK7y77rcdFd5a7OkrqqCbm2ZF6CL3W/bwXOn+OfUbeISe1kZLHV+59Rb667zbxLEPp5bm92TTMZA3cqjMUdsdFQmKP7SrscZc1kZNHWWv25jklRa66YRBU13vdJdbzLpbp+B8ud87Z5UrqA2jUnM5aOjKBOKv0y/rAyauqUUUm+tm0TUtNksJrRV6XL+uoq3uqpv3E0VtM7qj71n2FF8OcVfNi8rTd1yqbuoO8d1qYEuW+umMfQFTay8tttEhiyApvupXtd0D9Op/c5j1QpqOndx3pih5R+H5a+rG00NqNRd1FFharW7eKCmGbC2P7LqHHX74cbsz64bU6jm06YBcw2/Uz1TIeyXiOgb+vzDiX/b5UqkbN44c9KYR3QPczrt9/q6Gaq6H7lcIKdTKZeXzYWxreUfEtdnEqoB66p7Heo6jPbxL/4+liuRsrlbjstFeCSEheRS6Vw3fbb5kW1DW219/l37GHW/ictz6rsBsyXxbvzoxN+X5b22pj7P2lo31yc5PsXNtTK7Wrk2PRJb68Am2qcvcVL9JqZeWqoC2ldl9lTmRyf+vizvOnelS3RbKmV98LhWZlM3suq6ATJfftNQzCaCEpOlURe1lWIERIhufN2z8CggoxN/X/WhrtdqqueplvWC1POvpWnYpephNJ1/4CKSdb7K2HyMQys4bcRE9SxshN1jgz468XdtOHXl1Wb8qqtxxy4J1nOJQRh8WspthNdWGKpHQKFwyutY8B3yZ2MweGzQoxN/AJ8GcBHAawAerfu8j2ifJi4a3X4T1WNoBNGTWHxhPvPh+0HVRfiozh/6ucbyu/VJk8bP9rdUCftQLX8AEwCvA7gbwAqAFwHcY/pOyPX8655zeae5sYh/EE9CbL5pH5aszlJvOsnINrJINzCsup+292oa8B5jT8AWF+EP2KDHJv73AXim9P8ZAGdM3wkp/rb1S9dbG6LbJ4hOx+ab9sVi4W9J4SaWf13e2oqITYM0tp5AHYuF/rnZRj4NMdoHwGcB/PvS/w8CeFzxudMAdgHsrq+vN7rxOky/UdWwWSyOBncsLw+zzAfp6cdk+fvG17018fmHzpfvBmkM6J6ZEJ1HPsUm/r+sEP9/Y/pOKMvftlynHq3WBO/3OmTfsc9ejWu0T+h82TRIQ+jB+cTUW+qY2MS/F7ePSsxcJm/SsPHAUFvPWHs1Pnskxe/muqyFy7mHUiYiKg+xif8tAL4P4K7SgO9HTN9pK/4q40UIKY8dsxd/GjZES4hejQ9RDJUvX+ccam8wovuKSvyz/OAkgD/Jo3626z7fVvxN7h3VJM1IGu2kSN6Aa3sDxfcBt41XdOeqisfysnkV0Lp8+fxhbM5p85mILGTvRFIhohN/16Ot+JvcO9Op3V4SxUqxkfymURGRodMMXzfg6zw2g1G2E436KKy2z8FmTMLnPYyw8o5e/E11SQj9Am7lozDmquU1KZELRJIGXAg/tq8HYTsYZTpvn+FppudQfu5LS+rPFfHTfbuYBtBYjF7868JubepZ0zo4BpIL4Q8VwWI6j4uQ2IahmfLX58QUl81eTHn0aVW4nquv7qznBmf04i9l5rZRWe0+xD9akeuI5Cz/ELHrPib2lM9lI5Km/Jm+FxrTrGCXCuXTqnB1MfmOarIhQIND8c9RNapthT9qkeuI5Hz+IWatmib2NFn9s1xYp9NsCVmX/HUp/qrBbpWl5VqhurT8Y5jPEMCKovhrsIn1rzNWoha5DvHaWw3ta62z/Jtc01SQfE22cnkmXbl9TKJZ3HfZ1+/S6Hbp849hJnMA/ynFX4Pp9y5H9+j2pE50DChuuuhGmFr9ppXbZLX14RdbLI72FlZW2s0YVmHTkJbz5LqRfchon/KzsBH+0JYeLf/uxN92FvYABv3ToSuhNFnpTTA1WrEMHm5t+QtptRVNlxVIu8TWzdPl6qX0+fdv+Y/dh98rXYUOhfjxTcIWg+j5uGdb0Yy9MvmaS+EbRvt0I/7JDVSOga5a5KH/+CoR8dGw2vrGY3+edWM0bYQ3hoY+h+JvIKLfKRhJ3WOXohyzpd7m+q5xzS4Nq22EROwFLZSR0WRcIyAUf0v6ru8hSNLA7fuH6PuhuVxf5dfXCfTS0tFZv75CWmN17+jQRXIUkR5NicydRPG3oO/6Hoqh1NVOCf3Q6ho32+ubQtF0R1sr1LaidNGAt72GrofUJq8+lubwCMXfgqGKZHJLL8RAyIdmI56213fxv6sKdVPxrPteV+G6ba8RotLb/iYdVUCKvwWm+mYKD47dPTTURi0oIR+azbltr++yG5GqUIcS6C4KnY9rhGjkbaOhaPnHI/5t1viJ2T00VHdWUEI+NNs1Zmyu38byDynQXXQ3Q0Yu+Rj0LazD6bT9OEsLKP4Kqr+P7ZpTPTfkjeh7/DRJQj00F39+3fVdY+7LwhNSoPu0/KdT+9+tK8uoxwpI8a/QpM7UHfShEyt8D5hWP2fqwhbnWSzCrloZSlSrFptq+QpXK3vglhHFv0LT3nKqlj+JjJADpovFUVEEDjZyMVk+PoXSt6iq8l3d2tLHPIaBQfGv0HScrK4nHSsDN26GR1u3iUkEdeeeTOJwkeiweSaurqwRVAyKf4U2lr8QUn7yk+mUmb7rLKkhxDIMpu/brmZYpe+wMZtn4pLHEDH+EULxr9DU5x+70Kvou84SA7qW2cZvb8L0o+veK8I/dfQ9YcSmILuMp/he0jtSKP4KVLH7pgYh1TLRd50lBkwRK6bCWLehhEkEmwpf3/50nwPlpq7/wCoGxd+SxUJdxlPuDdLyjxibmYV13VHTmj/lwjydHnzOVfgWi6NRNEA2sNxlxfDloze5vgZWMSj+GnRlaUjjQPT5R0ybQcw6wTL98K4WgamHkiJNXV8JQvFXMCZRHFJjNihsCqGN9a+y2E0C71r4m/gOY14TRXX/Plb0jBCKvwIfEwQJaU2TmH8bi71OsF26va49BZs8921pjcQiovgrsI3177uMkoHQRmzKAmwbnthksEfXI3Dd+9c2lnpg/vUYofgrcIn1ZxkltZjE3aeP0WXZB9dr1rmKbHsKtpbVwCJrYoTir8Al1p9llBipE9q+Qq5cextNZsi6zlOo+lib5JNYoxN/kb0XP5ubm3J3d9f7eXd2gAcesPvsbAacPQvM596zQVJnYwPY2zuaPpsBly4BS0uZ3FURArh5M3Tu7LnjDuDq1aPp0ylw5crRdN19T6fAj38M7O+br7eyAnz+88CFC4c/u7oKnDvHyuYBIcTzUsrNavpSH5lJlb094KGHsgaDkENcvmxOX19Xv69LTwXdfV+7lon3bJY1cLMZsLZ29HPXr2efqzYS+/vA9rb//JKfMXrxdy1f770HPPJImLyQhKkT97NnM2u2zOpqlh4T1665pZvuez7Pej03b2Z/f/Qj9Wdv3FCn6xoW4oXRi7+qx1qHqlccEzs7WW98aSn7y55KIMoP+t13MxdGmZWVLH1pKbMyTp06bAnH6NbQifnx4+p0l0ZNd+7JxC0vxA+qgYAYj1ADvrbRPtUjVsY0ca1X6taW73nrvsY0WcahbfSRaxgpcQKM9jlK02Wd19a8Z8UbXMfHgTYRJnUPOuUfQhepM5m0j8bpYz2VtudOPBKJ4q+g6YYupuVM+i4nXMHTkrZdpLoH7fOHCF2oque3qQTVHbRiFcS2v/MAutIUfwVNLX/XkOcuy0nKBmentH1QXVn+NoWq7cxh1Ro3rpVC5xbq2xoK/TsX9H2fBij+ChYL9VanTS3/GIQ3hgYoCdpa5nUP2tcPUVeo2l7HtLpl24oRsjDaim3IHdHKeYm40gURfwBfAvDnAF7Ij5Ol984AeA3ARQCfKqXfC+Dl/L2vANlEs7oj5JLOtpMR68Q/FpdLxEZIPPhoqW0WZGv7Q9QVqrb3UbeuvRCZr9+2cpQJZQ25iG0Xln8MVp+BkOL/RUX6PQBeBHArgLsAvA5gkr/3HID7AAgAvwfgfptrhd7MxcUFpBPzyMsAKRO5tfYz6gpVW4vDptC6rINSJpQ15FLRuvD5x2L1aeha/M8AOFP6/5lc8E8A+ONS+q8AeNLmWqHF36V8u6xiG6OehCDJ3kYKmVYVqpWVg8FWnVVua3HYFtrys9KJXbVLHMoaarL+UMhon8itvpDifwnASwDOA7g9T38cwAOlz30NwGcBbAL4Tin9FwB8y3D+0wB2Aeyur68Hf0hbW/WuzjoxT0FPfDPmRq8TyoVKNX/AtZCazm9TaFVzAZaX1Q2GbcFwyUNsYht5BWgs/gC+A+AVxfEZAO8HMEE2U/gsgPP5d35DIf6/BODvK8T/d+vyIGU3e/jauH6Wlg7KWSS/be/EVhcHgU4MdQ/bRwy+j/w1+ZyreMYothFbfcGjfQBsAHglf52c20dK9wCHvstbLETu8kwPk7i1edixClQT6yHWe4mQUG6fE6XX/wLAN/LXH6kM+H6/NOD7RwA+XhrwPWlzrS7E3zXqp1xGx1z2aPl7xvRAmz7sLqzlpoIc2noYeUMRSvyfzsM2XwLwzUpjsJ1H+VwsR/Tkfv9X8vceR8+hnmWaiv/YewEx9sKTxiSGpklZJmEL3UK3KQQhXVksnOHdPqGPUJO8XGe11/UAxsrIjSu/2EzsKj5ju59vaOu6TeNiE2rXVLDZLaX4V/E1qz1EPSIDokmrqCucW1uHP+cibKFF0MeM6bpJZU3yWteLGoHFQvGv4HNW+wiNCWJDG5eDKu7YZXJRVdhCL5vss3Hx2UvR5Ws6HY07iOJfwSTyLrPZ68rNSIwLoqKNILZZVkAnbFtb4Qpjk4bONZy1SUOiy5dugG+AFhzFv4KpfC0WdoJfNBKm8OXqwnGmPTHIwGhjwbZZUKwvYXOxdEyNhe9BWlW+RhSfTPGvUFf2islcpqN6vmr50tVB034AZECEtvylTFfYbAe1Q3WZRzQQTPFXoCpftmv8TCaHz6NqSGwbDjJQbCxYnciFCJ0szh+DH7LvBmpEIaAUf0tsQz6PHTuoP03mB5io9hqm00GWyXFgEts6AWoq1LrzxrRXbgyWdywNYWAo/pa0Dfe0Oeq2gVRtMKNaN4skTkgBVAlbFxO9fPj8iVco/pbUWf5tG4c6ETddf4DuyHHTtesj5PV8RvsQr+jEX2Tvxc/m5qbc3d0Nfp2dHeD0aWB/3+95hQDW14GzZ4H5XP+5paWs5ujOcfOm33yRHtnYAPb2jqbPZsClS2ldr+t7IdYIIZ6XUm5W05f6yEzMzOfAuXNZmRUCmE4zQW7DbJaJ9qVLZuEHsgaiyXskQc6eBVZXD6etrmbpqV3v8mW3dNI7FH8F83km1DdvAmtr7axt17p19iywsnI0fXk5nCaQnqhaGrNZ9r/JQtjZyazspaXs785Os+sBwGSSdXG3t7PztDm3zjKhxRIvKl9QjEdXPv8qbXz8k8nh5Vhc9r9YWzvskq0u60JGQKglGlT++ZWVo7tz+d7rlvQCOODbjLarfS4vH4Rt2i7AuFjY7ZJHBozLyoOukQAuhdrl3BzAjRKd+HPAt4adHeChh4D33gtz/vJ42M5O1gNXjZsB2fjDlSth8kEiQzeAqsI1EsAUVdD23CQ6OODbkPkceOqprA6EoBgPK6KMTPX96tUweSAR4jJQ6upXd/k8ffaDheJvwXwOPP00cMst+s/MZpll7kpRt7a3/YeXkoTRiW7VCmkSraOK+llZyaIK2p6bJAPF35L5HPj617PonwIhgK2trAd96RLw2GNu5yzXLRtDr0njQhJFF5b5hS+4RQepUEUZnT+fdXHbnpskA33+Hih89Zcv27tSJxPgwoWDulXn4l1Zyeon6+KIKBcsmxmChCjQ+fwp/i1pOiO4Oo5mGlieTrNeBes9IcQVDvi2RDf/5ZFHmvnqjx8/fN4HHwRu3FB/dm2Nwk8I8YthCJMUVK37vb3s/z/4A3MEzmxmduVUz6vrhHGGPCHEN7T8LVBF4uzvA08+qf+OEMDJk/r3r12zj/BhtB0hxDcUfwt0lrdp7ouUWbCEjvV1O4ue0XaEkBBQ/C1oannrfPhAJui6804mjLYjhISF4m+BLuT62DHz9yYTdfp0mgm67rwXLtgvAU0IIU2g+FugW3n3ySfVyy8DmYifPq0W92IyWJMVfQkhxAcUf0vKa/wXFvl8nk28Ki+PDhyI+BNP1Iu76ryEjJY2ewoQJzjJixASB6oZk6ur7A63hJO8CCFxo4up3t7uJz8Dh+JPCIkD7gPcKRR/QkgccB/gTqH4E0LiQBf7zFmOQaD4E0LigLHPncKF3Qgh8VDEUJPg0PInhJARQvEnhJARQvEnhJARQvEnhJARQvEnhJARkszaPkKItwEYNkUMxh0ArvRw3bakmm+Aee+DVPMNpJv3rvI9k1LeWU1MRvz7Qgixq1oUKXZSzTfAvPdBqvkG0s173/mm24cQQkYIxZ8QQkYIxb8ewzbsUZNqvgHmvQ9SzTeQbt57zTd9/oQQMkJo+RNCyAih+BNCyAgZtfgLIX5ZCPFdIcRNIcRm5b0zQojXhBAXhRCfKqXfK4R4OX/vK0IIkaffKoT4rTz9D4UQGx3ex5eEEH8uhHghP042vY8+EUJ8Os/na0KIR/vOjwohxKX8ub0ghNjN044LIb4thHg1/3t76fPK599RXs8LId4SQrxSSnPOa9dlRZPv6Mu4EOJDQoj/LoT4Xq4rj+TpcT5zKeVoDwB/B8DfBvA/AGyW0u8B8CKAWwHcBeB1AJP8vecA3AdAAPg9APfn6Q8D+Lf5688B+K0O7+NLAL6oSHe+jx5/i0mev7sBrOT5vqfvMqLI5yUAd1TSvgzg0fz1owB+ve75d5TXTwD4KIBX2uS167KiyXf0ZRzACQAfzV/fBuBP8vxF+cxHbflLKb8npbyoeOszAL4hpfyJlPJPAbwG4GNCiBMA/rqU8n/L7Bf6DwB+sfSdC/nr/wTgkxFY003uoy8+BuA1KeX3pZTXAXwDWf5ToPzbX8DhMnHk+XeVKSnl/wRwrZLslNc+yoom3zpiyvebUsr/k79+B8D3AHwAkT7zUYu/gQ8A+LPS/2/kaR/IX1fTD31HSvlTAP8XwDR4Tg/4Z0KIl/Iuc9GtbHIffaHLa2xIAL8vhHheCHE6T3u/lPJNIBMAAO/L02O8J9e8xlRWkinjudv37wH4Q0T6zAcv/kKI7wghXlEcJqtSZbFLQ7rpO16ouY+vAvhbAP4ugDcB/KuaPAXNa0NizJOKfyCl/CiA+wH8UyHEJwyfTeWegPjLSjJlXAixBuA/A/jnUsr/Z/qoIq2zvA9+G0cp5T9q8LU3AHyo9P8HAfwgT/+gIr38nTeEELcA+Buw77rWYnsfQoh/B+BblTwV2NxHX+jyGhVSyh/kf98SQvxXZG6cHwohTkgp38y77G/lH4/xnlzzGkVZkVL+sHgdcxkXQiwjE/4dKeV/yZOjfOaDt/wb8k0AnxNZBM9dAD4M4Lm8y/aOEOLjuT//VwH8Tuk7p/LXnwXw33J/XXDyAlXwTwAUURJN7qMv/gjAh4UQdwkhVpANmn+z5zwdQghxTAhxW/EawD9G9qzLv/0pHC4TR55/t7k+glNeYykrKZTx/DpfA/A9KeW/Lr0V5zMPOfod+4GsEL0B4CcAfgjgmdJ728hG3y+iNNIOYBNZwXsdwOM4mCX91wD8R2SDNs8BuLvD+3gawMsAXsoL1Imm99Hz73ESWYTE6wC2+86PIn93I4vOeBHAd4s8IhvbeRbAq/nf43XPv6P8/iYyF8l7eTn/fJO8dl1WNPmOvowD+IfI3DMvAXghP07G+sy5vAMhhIwQun0IIWSEUPwJIWSEUPwJIWSEUPwJIWSEUPwJIWSEUPwJIWSEUPwJIWSE/H9eZaOb4ecC7gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pca_analysis(1,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACqCAYAAACTZZUqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAI90lEQVR4nO3dT4hVZR8H8HPfphJxLKzQippFZIOGTFEZFNEuxE3YIjAKFxYoKkqMItGqiCjEhSARWRkVgqCbyAKFslqJC/9ubKOgKP6JcuHoQPfdvBAvz+/Uvc2993e98/ksvzxzzsPM8cvB8zznNJrNZgVA7/0newIA05UCBkiigAGSKGCAJAoYIIkCBkgy1M7gRqNhzRpd1Ww2G70+p+uabqu7rt0BAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyQZyp5ANz3yyCNF9sQTT/Ts/CtXrgzz559/vsi2bt0ajj18+HCR7d+/Pxx74cKF1icHLRgdHS2ytWvXhmNvv/32Ips7d244dunSpS3P4dChQ0W2Z8+ecOy+ffuK7OjRoy2fq9fcAQMkUcAASRQwQBIFDJBEAQMkaTSbzdYHNxqtD+6huiet0WqBBQsWdHs6Xffdd9+FeTtPlvtVs9ls9Pqc/Xpdd8vw8HCYv/fee0X22muvFdmsWbNaPlejEf852+mddkxMTBTZ7t27w7ErVqzoyhwidde1O2CAJAoYIIkCBkiigAGSDMRW5E8++STMB+GBW2TRokXZU+AmMTIyUmQ//vhjOPaBBx5o6ZjffvttmE9OThZZrx/CPfbYY0X28ssvh2N///33IhsfHw/H3rhxY2oTq+EOGCCJAgZIooABkihggCQKGCDJQKyCgOkuehl6VVXV119/XWQPPvhgODZambBr164ie/XVV8Of//PPP/9uij0RbZNevnx5OHbZsmVFNnPmzHCsVRAAA0YBAyRRwABJFDBAkoF4H/Drr78e5h999FGPZ9Ib586dC/NWt5L2M+8D/nfqrvXo30bd9uAvv/yyyNavX19kV65caW9yeB8wQL9RwABJFDBAEgUMkEQBAyQZiFUQs2fPDvOjR48WWTsrBU6ePBnmn376aZH99ttvRbZjx46Wz9UOqyA6q1+v63ZcvHgxzO+6664i+/zzz8OxGzZsKLLopeW0zyoIgD6jgAGSKGCAJAoYIMlAvA/4jz/+CPMXX3yxyOq+oPzVV18V2c6dO8Ox0VbMuuNCpy1ZsqTI7rjjjnBs9JA9ethWVVN/4HbnnXcW2dBQXDHRvC5fvjyl89+M3AEDJFHAAEkUMEASBQyQRAEDJBmIrcjtqHtaPNUnwNH24Llz507pmHX2798f5i+88EJXztdLtiL/pe5Lxz/88EORPfXUUy0f95Zbbml57L333ltkq1atCsdGebQVuqqq6vr160X28ccfh2PHx8eLrFtfKe4WW5EB+owCBkiigAGSKGCAJAOxFbkd7TxsGxkZCfO33367yIaHh//1nP7OTz/9VGQrVqzoyrnoL3XXVDsP3L755psiW7lyZTh206ZNRXbPPfe0PK923HbbbUW2Zs2acOylS5eK7J133pnyHPqBO2CAJAoYIIkCBkiigAGSTLudcO04cuRImD/66KMdP9fExESYR+9YnZyc7Pj5+4WdcH+59dZbwzzaCfnss8+2fNxGI/4Vt9oFhw4dCvNjx461PIeXXnqpyOp2qZ4/f77IHn/88XDshQsXWp5DL9kJB9BnFDBAEgUMkEQBAyRRwABJpt1W5DrRE+dou2QnRCsetmzZEo4d5BUP/L26v/1bb71VZAcOHAjHRtf11atXw7HRl8Hff//9Ijtz5kz48+2IVm1EK36qKn4n8UMPPRSO7ddVEHXcAQMkUcAASRQwQBIFDJDEQ7j/id6FOn/+/K6ca86cOUUWfaQQIj///HORLVy4MBwbfYDz2rVr4dhOPFxrVbTtuW4r9OXLl4vs7NmzHZ9TBnfAAEkUMEASBQyQRAEDJFHAAEmm3SqI0dHRMO/Gl4brvsDczkvwoRW//vpr9hRCdf/e7r///paPcfjw4SI7ffr0v55TP3EHDJBEAQMkUcAASRQwQJKBfgg3NjZWZHv27AnHjoyMdPz8b775ZpjfuHGj4+eCfrRz584wnzVrVsvH2Lt3b6em03fcAQMkUcAASRQwQBIFDJBEAQMkGehVEM8880yRdWO1Q1VV1YkTJ4rs4MGDXTlX9KXbqqqqhx9+uCvnO3/+fJFduXKlK+fi5rVhw4Yie/LJJ8Ox0Xb8HTt2hGM/++yzqU2sj7kDBkiigAGSKGCAJAoYIMlAP4TrpeirtFu3bg3HHjlyZErnqtvGuW7duikdt873339fZMuWLQvHTkxMdGUO9I/nnnsuzLds2VJkjUYjHHv16tUie/fdd8Oxk5OTbczu5uIOGCCJAgZIooABkihggCQKGCDJQK+CiLbQRk9fq6qqhoeHO37+pUuXtpVna+d3MzQ00JfOtDRz5swiW7NmTZGNj4+HPx9tL65bwbBx48YiO3PmzD9NceC4AwZIooABkihggCQKGCBJI/qP89rBjUbrg/vUL7/8EuZPP/10j2fSG9GDyKqKt33Wje3lV2mbzWa8d7WLbrbrevHixUV23333hWOjv90bb7wRjl27dm2RLViwoM3Z/b8PPvggzDdv3jyl495s6q5rd8AASRQwQBIFDJBEAQMkUcAASabdftJXXnklzLdt21ZkdS+ernsheq9cv349zA8cOFBkdU+bjx8/3tE50Tvz5s0rsi+++CIce+3atSK7++67w7Gtrog6depUmEdfNf7www9bOuZ05Q4YIIkCBkiigAGSKGCAJNNuK3I76t7bO3v27CJbvXp1kW3fvr3jc6qq+i8P93LLcLfYivzPxsbGiqxui/2MGTOKrO5LxdHXuqNrKnrYVlVVdfbs2TDHVmSAvqOAAZIoYIAkChggiQIGSGIVBH3FKggGkVUQAH1GAQMkUcAASRQwQBIFDJBEAQMkUcAASRQwQBIFDJBEAQMkUcAASRQwQBIFDJBEAQMkUcAASYbaHH+pqqrT3ZgIVFU1knRe1zXdVHtdt/VCdgA6x39BACRRwABJFDBAEgUMkEQBAyRRwABJFDBAEgUMkEQBAyT5L95QHHzClR5NAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAD4CAYAAADCb7BPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAArd0lEQVR4nO2dX4wlx3Xev7PDGcZDOoE4pIQN5Z1dJkIQ6iUQN4aIIHrZIPzzojimANJjiYEFLDBy8pBAD7vYF73wQQqCwAodR4xFQdFdWEoCBGIcK4wkw7CBBKaHgSguE625FLkkLUIkLcegTMCydysPty/Z09P1v6qrqu/3Axoz03Nvd1V39fmqzjlVLUopEEIIIb4cK10AQgghbUIBIYQQEgQFhBBCSBAUEEIIIUFQQAghhARxQ+kCxHLrrbeqkydPli4GIYQ0xdNPP/2mUuq2mGM0LyAnT57EwcFB6WIQQkhTiMjV2GPQhUUIISQICgghhJAgKCCEEEKCoIAQQggJggJCCCEkCAoIITouXgROngSOHVv+vHixdIkIqQoKCGmLqYz6xYvA2bPA1auAUsufZ89SRAjpQQEh7TClUb9wAXj77cP73n57uZ8QAoACQlpiSqP+8st++wlZQyggpB2mNOonTvjtJ2QNoYCQdpjSqD/yCLC9fXjf9vZyPyEEAAWEtMSURn1vD3jsMWB3FxBZ/nzsseV+QgiAGSymSNaIlfG+cGHptjpxYikeuYz63h4FgxADFBDSFjTqhFQDXViEEEKCoIAQQqqECwHUD11YhJDqWM0ZXU37Wc0ZBejBrAmOQAgh1cGFANqAAkIIqQ4uBNAGFBBCSHVwIYA2oIC0CiOMZMZwIYA2oIC0CJcaJzOHCwG0AQWkRUpHGDn6IROwtwe89BJw/fryJ8WjPpjG2yIlI4zMrySEdHAE0iIlI4ylRz+EkGqggLRIyQgj8ysJIR0UkBYpGWFkfiUhpIMCUjOmYHWpCKPL6IdBdkLWAgbRa6XWYLXtnRy1lpsQkhxRSpUuQxSnT59WBwcHpYuRnpMnl8Z3yO7ucsRRK62Wm5A1Q0SeVkqdjjkGXVi10mqwutVyE0K8oYDUSqvB6lbLTYgDDO8dJomAiMjjIvK6iFzq7btFRL4pIs93P9/T+995EbkiIpdF5J7e/rtE5Nnuf58XEUlRvibxSdWtqVVzESMyU7iC0AhKqegNwEcAfAjApd6+zwE41/1+DsBnu9/vBPAMgBsBnALwAoCN7n9PAbgbgAD4BoD7bOe+66671GxZLJTa3VVKZPlzsRj/zPa2Uss2vdy2t8c/OxUu5SakMXZ3Dz9mq213t3TJwgBwoGJtf+wB3jkQcHIgIJcBHO9+Pw7gcvf7eQDne597shON4wC+19v/EIAv2M47awFxofVWXaHYlChShZeBDBAZf9RESpcsjBQCkjMG8j6l1GsA0P18b7f/dgCv9D73arfv9u734f4jiMhZETkQkYM33ngjecGbolTQOoXbrEKfQIkiVXgZyAgM7x2lRBB9LK6hDPuP7lTqMaXUaaXU6dtuuy1p4ZqjRKtOZfEqXFerRJEqvAyTUVP4zgbDe0fJKSA/FJHjAND9fL3b/yqAn+l97v0AftDtf//IfmKiRKtOZfEqTPktUaQKL8MktDby4jtKjpJTQJ4A8HD3+8MAvt7b/6CI3CgipwB8AMBTnZvrLRH5cJd99Yned4gOn1adyu00NlEQ8Ld4FfoEShSpwsswCS2OvPiOkgGxQZRlLAa/AeA1AH+B5UjikwB2AHwbwPPdz1t6n7+AZfbVZfQyrQCcBnCp+9+j6GbKm7a1D6K7kiJba+wYMYH7CjPIShSpwsswSupA/9yC0q2BmrKwSm0UEEdis7UWC6U2NvTiEWrxKkw/YhbWUXKIXOsJhK2TQkC4Fta6cOzY8vkcIrIcj5sYLpA4xmLB8fyMybHE2Viz2t5mXGEquBYWcSfG0T7mrO6zu8snfubkCPQzKN0+FJB1ISZby2QlSuQxtpT7ORNyBfoZlG4bCsi6ENPd01mJjY3pu4yt5X5moIR+cg4EGSU2iFJ6YxB9AqZKE3KJJK955LVkxlbtgX7iBxhEZxB9Mi5e1L+FMNXxXSKqMckAM4Dv6yKpYBCdTMfKWf2Vryz//vjH0/pPXGeVreusu451nbVug2GxMlBA1oFUT1fO+IPr7PY1dMb3b98xzRO7Jvo5Ss1hsdkLW6wPrPTGGIiGlcN6NbU3hdM8Zfyh71Df2Rk/ru7Ya+SMt03+r3XWuo2Ut1DXLDc2yr8Wp+YVBsCZ6GskID5PnIvVCTH6qdaecCnf6ri1PG2FMBnHVvUztWHVNcvSBrv2fI8UAsIgegv4TtnVRVr7hASdU0VwXcq3ovH2GcsccwZSJwLYmlOpBIPa7x2D6OuC77KlLhHVEKd5qviDa8R3d9fvuJHU6K+eImdg6nqnTgQYa5Zjx526nmuR7xE7hCm9rYULy9d1pBs7pxjXp3Be28pXwPdQq786d7lK1DuHa8e01ueqmXKV5cOAMZA1ERDfJ26x0ItO6cjiqnzDJ2traxlML+TYr9lfnTNnoES9cxlW03FNfZacza3mfA8KyLoISMgTZwpMu5wvtNW7freyJyv3uykqq+47lHonR67roTuuKdBe28hgKigg6yIgSvk/caFdy9Du4WIxno7byJOZsydesyuj5pFXSly8pnOrsw0KyDoJiC+hVivEoqR+U2H/uBN123Ma+ZqNdKoXVfan9KTwRKa+9S6Z4+v2JkQKCAXETMhTGOLTsHXvQp7MAt32XHpV+6tbfes9FIzNTf2tD7llOWMktljIOkEBoYCkJ6S7bHMwhzyZNXfbPZlRVZzngMbUM/f1qtmlOCUpBITzQFyocYJALkLmepgS20PXqapk1cAUt35Oy3fZXk45hu8ty33r+SbEhMQqUOkt+whkHbsrIT6NsW7pzk74dco1WcCjXilvfW1ZWKHlsQ02WxiBkCWgC2sCASnZmmuzOiamiHrGToD0PN5cDZnuUuzvh7/Pq1QMxKXZpWiaLT2KrlBAphCQkony6zbyGZLyqQ1Qg9qD36HoLoXLos22OaBTZmEtFvoA/uo7qbLM5vgoUkCmEJBc3VDbE6I7785O3HnXlQA1mOsIxMcNNVbXWnrjprcArIy87jM+93Cu7YACMoWA5Oh+uBzTJbOp9S7QlARYgbn2PH3cUDWPtlzrEFuvuY5EKSBTCIhS6btcLsbM5SkXWTquY6ilO5mbQDWY4+UZuxQ6I1nzCCRGQDgCoYBMJyCpcenSLBZuT0LMS5fm2sXWEWv5MlrOqY3y8Hz7+25NoaYmY3NhAcvPMAYyDgWkVQFx7dK4PCGmrlBonKX1rlUOMlqRWgyUi4jV1GQWi2UAX/dYrK4hs7DGoYC0KiCuFmOxcIt4jjljY+IsrTt3c5DRctZklG3kbDIhRnq4rErBNwI0RxMCAuAlAM8C+M6qwABuAfBNAM93P9/T+/x5AFcAXAZwj+34TQqIUu5Py/6+XUR2do4eKybO4mq5QrtlKbtzCY9lPFRGy9mSjrs2mRRzUefgJqqZlgTk1sG+zwE41/1+DsBnu9/vBPAMgBsBnALwAoAN0/GbFRAfTIKwtXU0Gd60WNEwzhL61IZ+N/X07kTHsh6KIxCllNslD7ktLV2DudCygFwGcLz7/TiAy93v5wGc733uSQB3m47fnICkDuTq4iSm93umKE/oE5/SUiQ8lvVQaxADcSVHaK2lUdhcaEVAXgTwvwE8DeBst+//DT7zJ93PRwH8Ym//FwE8MHLMswAOABycOHEi7VXNSQ5LYXJvuUwtTn1e2xOf0lIkPJbpUCuD+QtYqFc2dtV1tJ+F5UJomVK+EaD2EUiN982VVgTkr3c/39u5pz5iEJBfHRGQnzcdv6kRSI6nxHVdihRzRlzOO7MRSIo00KlIHVryrbfJ09q/LWPlbG0UplSbZe7ThIAcOhnwGQCfXlsXVo5xeuyssJTnnWEMJMVSGFOQ2pj5avPY+cfKYipna735VkdNK6oXEAA3Afjp3u//E8C9AP7lIIj+ue73Dw6C6N+fVRA9V4sbPnm6p9hFqHye4jXIwnLV/JhipKhC6qbl29exjTxWdarJ6MZe99bjNi0IyB2dIDwD4DkAF7r9OwC+3aXxfhvALb3vXOiyry4DuM92jqYEJMYv4NPKQ5/S1sfkGXC5lCWS2YakNma+Tcj1/LbPTTUKSXHdU4vh1COw6gVkiq0pAVHKrZX0nckhgfDQpyPkicjV6ivxZ7hcyhhDksoI5TBmPk3I9fymz03Zf/G5XrqmWKk31hkKSIsCYsPmTHa1CiEG2Lcbm6vVx3bpEwuP7ZAxvX9TEp1PmVzXsvKpn683M3YtrSndWz6uSVO9TNfI5/qVcO1RQOYoICZnso9lSnluXSvO1eobc8HlGIGY1sjUVdPljYKuxwrtRbucX/e5KWMKKUZMJnyva4l4CgVEzVBAXNa+ytUtqaXVhx63UIQ2xgjv7/vf5kozoaNJURYfEXPJBjM9hiF10b0PjiOQQtvsBMQ2Asndo65h3K057nVAvbKxq35vP17QUnu6Qo5n81bq9LLSuZjRjF2PVflcrqmvkI/dMxcPMrBc6MGESXx0YU/GQCgg8cQ+RVOS0pdiO25v+zG2x0XEUdBqSTaz9RXmPgLRGXCX/JGx78bUxfR93WbCdCxdeZiFVWCbnYAoVUf32PUYKaO5I+e7rnkKX9nYHf+Ow7lrcduYeqm2nnPL2T8u5w3J1tJdS9toynXU4dNWFovw8kwFBSRUQCpJEZ2kHCEWIkYUEltnnYBcgyEzzHJNndw2E9wb3aXa2LCfbn//3fUyNzbiVqkp8TiYYgSm3rtprqzr+qGuZQkR9z61r2JAAVEBAlKL/2KqcvgadJMLzeUYKZ3qi8VSKEaONzoCccR6SSa6N6GnyV28KQTFJVdEd49sIzff62Iry9ZW2IuqajE1OiggKkBAavFf+KZphJJyTQqXY0zgoL8G0QfSHbA+2D51iLS2IV/P2YRL92tMm8t8kZTXc9XEfUZ3Y4P3GpwdY1BAVICA1JJ24pumEYqvtfHpGo4c4/f2F+rPJJEF0pTlOuB/rAFGQ5NqllmmMqZuwv3z6NxAGxtpjaBv3GEYdkt52W1lcRXm2kccQyggaoYjkNRl8W3VunI5LKmyOtVDWKgXsauuQdRVMaTd2ih1r1zPm7F8ptuW8rQhAeRURnGx0AuWrV458kx053cV5lpMiysUENV4DCS2xfqcy3VcHZGam/wBqjVFaEXG0WxIFtKweC5GNsSVlMoomh6BEo9mbPutxbnhCgVENZ6FVSJNY3/fPpoIvD5ZHqBS9yrG+ia4f7Gr1sZqoG2LNYouIx9TtniIWIaUyUfAOAJpcGtiHoiudS8WSm1uHm5tm5v50mnGxCNhKy/1AK2qmfOVs9oTZxohhVxLl1jG8Pu6PozNtRR7T00jn7H+jG1BQ5dlSVyaRIwQ1eLccIUC0oKA2Fr31tbh/21tpXMwD4+ds0tpqWou+nGXH6OQuyvDCMn3WrrGMvq3eaz/0m+CJhFxHQH4JgKMHdsmpqaExhLtsQbnhgsUkBYExNT6c3bZdV3LXF3KjqkfoNUlfBG7WevlQo7AruvxXGMZ/cthMrxK2Rd6tI0ATPNPfZq+zZ3n64ar1aXkQ4q2RgFRDQiIqfXnjLr5PFEi4U7jwt2t1SXUTTicKoJZ2n3hYkSHt9ml+fVnvA83mxiY3Gg+1yt0BGK6Di2Tqq1RQFQDAlJqBOLzNIWsg1HaYnbUMgIpHUB1NaKhZTZ91ncEMJYIsLOjn+0dGgOpfSmRUFK1NQqIakBAbDGQXEZY9/TcdFOaUUNpi9lRPAbSUTqF0zWryfYdXUKeSQx0/z92LH05fLKwKunjJCdVW6OAtCAgSplbfy43UO4Mr9IWs0fOLCzX21ODnvaNvcO8z0Pfce35j9VPl6+xsXG0CY6VI+baTZHeWxscgdQoIHNqaX1LsnJEp65TDRYzMz492Np6uymas80tthKp3d3lwHbsMzffbC9H6Ko+tV3zqWAMpDYBmVNLzFGXdfIP9PDVyDn1QZQyG3afuIftOvjMCXH5Xsk+zFRtgFlYNQlIjS0xBFPyfmhdbDGcOVnMAbm8dK1cNt/sKt1ma3o2V5nu+773x/UdKqH3x3fEWroNUEBSCUhF/vxgbE9haF1MVqR2CxiJa7/CxxjUPHAb1uPMGT+hMG22a7NY+Dddn36fbk7LUERi7o9Pe6mhDVBAFEcg72BzWIfWxcVXkWr2fGW4ZgelWOzY5/a4CJZvcHls0p/u1vuOQMauzVj5QlyGsVlkGxvp7o9rP7QWc0MBUYyBvIPJ0MfUxXWCQeoXYVWCzRD7GoPYwW4KUXPJrrJtoYswrpqK78z1kPvjWs9U98fUFvrlNF3TKaGAqBlkYaU6r8nVFBtAd7U2qajBQeyIr8GJ7X26fN/2Gdc+gW5bjVhiRMTF0NomF9qaiEs9U45AdMJ95ozbteIIpMDWxDwQHSlHPjlHUS7dp1QCMuFoMHeqq851FFM9F8FKuXaUaT6JzT3kuw0Xeoydf+tSzzNn0t6fMdegSzkYAym0VSEgoZZI9/SFuoOm6LnrZrincmFN5CBOpVO2AdrYMWNuU8wIZJX34BrDWLmVQnv6OleVyxIjpnq6NhEXcRtrVikfI1sZmIWVeANwL4DLAK4AOGf7fHEBibFELjOoanPnLBazmOGeUqdsPfGU2ucaA/FZyX9MZFKlr441X5c6pFiD1MX7mjPuYMos820XOczA7AQEwAaAFwDcAWALwDMA7jR9p7iAxFgiF/+HLtpYUlRyiprn9QwtSg6dmiob3CXDauw9H7YtJplusTg8stjZCQ9+K5VmBNI/zxTiPjyvLas+hUDHMEcBuRvAk72/zwM4b/pOcQGJsRq25Hddy3dd6KhFPJ6WmAcrh6eslvTM0LiEbSDpm/EU0yxTxEBcjxeD7pqY7oHvAti52tUcBeQBAL/e+/vjAB4d+dxZAAcADk6cOBF3FWOJvbsmh7BvmksNLi4fdE+f47AiR8aMy+UzFbuGbPCY7CjdtbPVLYeRM41qQkaeqQfOpmsSur7XGLlGtnMUkI+NCMi/MX2n+AgkRdqG7vshXcmYLK4xZ3UuV1UCa5tiTkWIEbLNsXA5ZgkvoMvmm368Egnf4/XxEeT+Ao4x1yzF9U/lZos5TwxzFJD2XFhKxbfGvlj0V8/1mR4c07LGntTNzaOR2JTd6QRPRQmXUYpz5h6pjB1/a+vwnArfly3Zmp3u/7brEtN/Kp3xburApLzHjIG4Fga4AcD3AZzqBdE/aPpOFQKSAteA+ZiohHT7Vuf07a6mss4JxuUlXEYp3Ak5MsB8vYC+186lmYSE5mI9uCV79LbjpBxlMgvLtUDA/QD+sMvGumD7/GwEZOrUEpccx1hLmaq+lmpMmZBmLbZDgVL5tFN4T12vnSnfY2yzZWHZjmnKIYltjrVc/9LMUkB8tyYFZOzJDWnVY9NcY9OQcnT5dNegwadvrNj/ZHOh3trZffd+Weo0VQ84NbrRwtjmcittfSCXPk7pmMLUHZiUUEBUgwKiM5w+TulhekpfbFzzA0NSdVIb+Eafvv4AcE9G3sVuuYdT+OBD6mNzg/muf2UzyD7zaMc+EzpvpdG+S3IoIKpBAdE9Dbp1H1yc1yHdKNcRSKq0l8rx1bLVbXgRu27XMMH5hp/PGdDXrYzrujCgi5CZHoWxcqZcAMHl+udOQizdd6KAtCggttQNW6uyGX6fALprhlfOtKYKCOmRrm7DNWSK9DqUz2f5c13T0jUn04sth8fyzeiy1cun2ZvOEWOkc45SahkBUUBUgwIS2220df98u58uju2cCwZVQMgtWd0G6wgkgWUwlc+1Jx0y4c21KegGxa6BdBcj7+uuCzXSKfJTbJRIPx+DAqIaFJDY7oepdfsaK9dMrJm+LGpFSCxhdRsewkgMJLHbLzbWYTJYISOQMXR9kVQ9a1+jGzpimWLxxanWTLNBAVENCohS6cfWrt29Ia5xkJSr7VZIrLF5CAv1InbVNcgyGyvxtXItn65ZhUx4C3k7YM6etW+/K6ZTYOtLxcYuTLGfKeMiFJBWBSSWsfSYkJY397W2HIl1d+gWFwy5JWPfcymf6TM2w+6ahWWrQ+6etU95YtySpn5UisUZxu7V1tbRJIHccREKyLoKSJ8Yl1jIXJCZ5jumzIqJESTTCrSm8pkM5v7++P98VoR1wSerKjcxiRG66xiaLKArX4pEhBgoIBSQOL/BWGJ/rvUj1ojQWxJzK029/9xBW1vgWecBzZ3KGpqarROdnCOsEnERCggFJLzl6dJ4z5xxiyTWkMheKaG3JMaImEQii3Hq7v91iLoqu+ohLLz6HLWksg4xNeucQlwiM4sCQgHJ091dLMwCMrbV8PRXQokRSEwMxHbcIwZ15GQ/xrZRRIZiVUsqqw9zmxtCAaGAhLc8W7c0JD4y1dNf+egnRwzE9by6YHjK8ryz7tdgexG7zk2jllRWX3I2vambNQWEArIkpOW5pOb4rtY7xdNfq+9jQMosrBTfS9lEdLPvr0GcB6em+SeV3crZQgGhgITjmhvatzqpRyA5hG8NCdFUl9Rd3W3Wzb5fjUCGL6/SBdB1/RPXzLPaaK28FBAKiB+x80dMIrIKrPuUJYfrbQ3x1VSfyYNj2z/bOXqAP5Nt9QtYeI+adDPeXdcWrYVGBsaHoIBQQNxJ0cIXi6MzqVab76SCEpHmGTDWyzWNFsb6Byb3kU083mkyibrbIXNZa6TFZkkBoYC4k6KFLxZK3Xzz4e/fdFOY8UiZflxhVy+HO0NXdd0kNN27rXwMdv9YOdwyvrkaoQPN3O6lFgfGFBAKiDuxLVw3+ghdJyt5bmk95NI400xvn5X5XUYaU/WifUUxdNZ37j4HRyCNbhSQEcYMrE8L739/Z8e+5HutT3Uhci1L4fMqGVMP3nQrc85z0AXt+9drtSZoyuYxhXFvsTlTQCggR/GJkIa88TBmFDNW1opHEiGYXESx7gwfQ2j6rG0OaewtGcvVCGmSqZrHVO6l1pozBaQ1AZmihcVah5onEDaA6fLFXiafXq7pszl7y2PH1hlw33eOuJzbJ2lg3ZstBaQlARl7sjY37QnzvoR0t1yS/3WbaeZXbsGssMtnuoQ5XUG+n8116UL6HylGBqUEs2UoIC0JiMuTFdOqTXEOU3crxGXV33Rrded+aqc4foCFNQW61wGfPkjKEYhtlBErmBX2VaKhgLQkIK5PVqqAtKthje0y6rqLuf0GOY8fIU62nnDLRijGAzqWUhzy1kMdOeMccx3BUEBUQwLiaqhDWrxt5KFzkqfwN+gMdm5fTk6LESlOY4Z2MiPkYOVDhMy1/KYcDl0WVgpRzdmfmGsMhQKiGhIQV1dRSKv0NaYuZem/oHlnR6kbbjj6GdMcEJM45ZwUkeKpziBOkxghBysfKmShWeC5R1r9fpBu4mQsLU4SdIECohoSEKWOzq9I8YJlpfytk23kMSxHiGPfJlJTpiT5HDNTGlVsboOTIXZoB6GxghqNqCnja4rEAI5AKCBlSdVV8zWmtsWTht+LWXZEd55UzulUXd2YOJIDvkYoSB8d7pNtMmKOl1LlYqoyMQZCAZk/PsbUtxsaM7U69imfyh8SEkfywNcIBV22yBGI6X81GtEpR0WtJ0CMUbWAAPgMgD8C8J1uu7/3v/MArgC4DOCe3v67ADzb/e/zAMR2HgqI8m/dvknzm5vh7rYYyzOl1dJYo2uQZAbD5zYFGcfIGIjtnLUZ0RpHRS3RgoB8emT/nQCeAXAjgFMAXgCw0f3vKQB3AxAA3wBwn+08ay8goUZWZw1M8Y5Q6xEicJliEVo051u9JGnq3nawcYzIwmrNINc4KmqJVgXkPIDzvb+f7ETjOIDv9fY/BOALtvOsvYCkfupLR0tdMsRiy+KYZ/tjbKuHsEhmSH10tIRxnPqcKUY0tY2KWqIFAXkJwHcBPA7gPd3+RwH8Yu9zXwTwAIDTAL7V2//3Afym7TxrLyCpDb6p579ambfEWl7DcoTiMNPvGkS9iN1D4jG8pCm8hrZXv5YwjlOdk6OH8hQXEADfAnBpZPsogPcB2ABwDMAjAB7vvvOrIwLy8wD+7oiA/FfNec8COABwcOLEiWwXuAlSj0Bc56vonvhYC+QyY39rK+uMM5ccA1/j56KLIS+IbLX33Zq7bI4UFxDnkwAnAVzqfqcLKyUl5kPonvgUZQk5rw8OIzZbNUKMX+qVbFrvwZf2lJLKBQTA8d7v/xzAV7vfPzgIon+/F0T/AwAf7gXR77edZ+0FRKl8XVEXq9d/4lN0K11HQKlddIMymi6p6bL4nja0Wq334Fsv/xyoXUC+0qXkfhfAEwNBudBlX13uZ1p1cZBL3f8eRYtpvC37FYa4WL3+E5+qW9m/hjleGhHZddddltVkPNfTxlSr9R586yOoOVC1gEy1VSUgc3sqfGdn5+hW5nTRBYq8ac6Eqar90+7sLKfXhFZrDj34OfW1WoQCUpuAzOGpHrK/P24tVy+v7pNLQCu0NLZRhOs8yZipNXPqq5DpoYDUJiCt+xXG8BXFCo19DmzevSPGPMN1qfFS11gmMg4FpDYBmeMIJGYhxZYtiaX8LjGNd277mgwX1qSas4ECUpuAzPEJChFF23WoXVzGyj+yTrgt2/kdjZ1jx2KENanmbKCA1CYgSrVhHGOnUIfOmludz/XVdqWuo6d/ymo45+jaHGFNqjkbKCA1CkjN6Iy37n2j/e/5GHNTl9ylm1p6JOcy/6VXXmtx16RrvibVnA0UEAqIH6YJDKmMtS3H1aWbWtoSucx/GXSrjRobIYg27a5pwFta94kfFBAKiB+u62nEGGvbLDvdy6n6CySW9oV4Rcg9julp6V1CSbUZ7JoEjZihgFBA/HBdT6Nv8H27v6ZjKuUmIKVHIP266UZoNrdfAmyXYarLRFGYJxQQCsg4uifedT2NlUH37f4Op1aPWTWX0UXurnVIIkH/8/v7k3T9bZdqioFajaMckgYKCAXkKD4ptLp1plZrbYR0f02jGdP3ht3m/f13y7exsfx7iuvjwkRd/xpGIDUMBkkeKCAUkKP4PPGmmEho91e3rRgz4MM3K+Xq4S8WaRZnTN3114yIaoiBlA5HkXxQQCggR/F54k1iE9r9dTHOtlUFQ1YqtGFz3/lYxJhuuac7rHQWFkcg84UCQgE5is8Tb+rChnR/NzeXowmfLrGvKyz1dQmxiKFdf9MM90otNGMg84UCQgE5iu8Tb+rChnR/fbvEU6QW287jahGHIyff98NPJZaJYRbWPKGAUEDGmfKJjz3XFJMbTedZBehtdUjRFc8oljTyxBcKCAVkWnKks4YurxJSdt15XOqQIhiQSSzpZiIhUEAoINOR03/fn7S3ypIazl9JISZjx3EVhhTpSJnEkoFuEgIFhAJymNR+DJc5I6n892NvPvQZIYTiKgyprHQGXxNTbUkIFBAKyLuk9mP4zFpPYVR1VjDFvA0TrsJQsZ+IIxASQgoBOQYyDy5cAN5++/C+t99e7k91vDFEDv+9vQ088oj/uZQa/9+1a+P7X37Z7xw6HnlkWeY+Y3XY2wMeewzY3V3WeXd3+ffeXppyROBaBUKSE6tApbe1GIG4uD1S+zFcMoZSBbtN58o9AlFqFilM1irMoI4kLaALaw0ExNV1ktqPYZuzIJJufSpTdtJECxe+wxwNbcXuN1IOCsg6CEgpH71LDGRjI40R0mV4rQQqxqj7fHeuhpZBEjICBWQdBMTHNZUrC8vmxkolIql7/r6CEGJoY2enTwHTtMgIFJB1EJCpe49jhly3tHsNPVmT8PheO1MsxnV2eo2jF45AyAgUkHUQkCndKrpz3XSTWUBK9WRt18a3520abW1tHU0YsAlrLUa6YtfcHENOrUABWQcBUWq6p8xnsb/SRnKx0GdorWIzvj3vmLkvtYnrkAotdcW6thZQQNZFQKbC9yVRqZ54X+PmYuh1dbGVd7FIKyA1jEAqhZ61sqQQkKiJhCLyMRF5TkSui8jpwf/Oi8gVEbksIvf09t8lIs92//u8yHImmojcKCJf6/b/voicjCkbCeDEifH9OztHZ6qt9sdOprt4ETh7Frh6dWk/rl5d/n3xov47LpMclTq6TwR4+GFzeVNODORsPiO6uaCp5oiS/MTORL8E4B8D+N3+ThG5E8CDAD4I4F4A/1ZENrp//xqAswA+0G33dvs/CeBPlFJ/E8C/BvDZyLIRX3RTmn/lV47Owl4sgDffjDe4ITPoQy2MUsBv/Zb9czs77sfc2Xn3uuzsLLfKZqrXiq6/ottPKiR2CLMcCeF3AJzu/X0ewPne308CuBvAcQDf6+1/CMAX+p/pfr8BwJsAxHZuurASM7WvPCTFNDRW4xqTWCzcX7PrOpmywhhEaRgDKQtKu7AM3A7gld7fr3b7bu9+H+4/9B2l1F8C+FMAHl1BkoS9PeCll4Dr15c/c/egQ7qhYyOl2PP12dsDvvSlwyOuW24Z/6zLiCbETbcGVLy8GHHEKiAi8i0RuTSyfdT0tZF9yrDf9J2xMp0VkQMROXjjjTfMFSB1c//9fvuBccuzv7/8CRxd4HHF5ubyuCdPAseOLX/qjPhQSH/0o/HPDd1pFy8ePv6nPrWMu6Rc6HJGTN1fIYmJHcIourBIDDlScXSTHzc2lvM5QnwmLuX0TQOuJcWXrCWo2IX1BIAHu8yqU1gGy59SSr0G4C0R+XCXffUJAF/vfefh7vcHAPx2V0nSAsOet6t7Jkcqzt4ecPPNR/dfuwb85CeH97mOBFzWTHddAn8Fo8WkcWLTeH9ORF7FcnTx30TkSQBQSj0H4D8C+D8A/juAX1ZKrV7ssA/g1wFcAfACgG90+78IYEdErgD4FwDOxZSNTEiMjz9XKo6PAF296iZ8P/VT7/4+lsLsc06m+JI5EDuEKb3RhVUBMW6oXKk4oZlaY+eOXVJ/zJXGVCNSGFTswiLrRIwbKlcqzpjLaXMT2Noyf2/MpeU6V8UlO2x7G/jylxktJrOAAkLiiXVD5UjFGROmL30JePzxd/fpGAqfq0CassOG4hgaMyKkIkQ1Hqc+ffq0Ojg4KF2M9WYVA+n30re360/qP3lyGf8Ysru7FDLfz5m4eHE5Ynn55eWckrfeOhzQb+F6kVkhIk8rpU7bP6mHIxAST6szwlwyq3w+p2OYZPDHfxyeDUZIRVBASBpKzAiLdQO5Cl+sQLqm93IVQdIYdGGRNmnJbXbs2PjqwENsLrG+G+zEieUIqLa6kmagC4usLyGr+JbCJZlgzCXWH2HdeivwS7/E9bRIVVBASJu09DIJXUqxael3xk1IA1BASJu09DIJXUrxm2/qY0aMm5AGoICQNonNjJoa3yQDV2GoUTDJ2kABIW3SaupwH1MWWWjchJAJoYCQdmn5ZRK2BShD4iaETAwFhJAS2LLIQuImhEwM54EQUgLd3BCRpUAQkhnOAyGkVVrKIiNEAwWEkBK0lkVGyAgUEEJKMIcsMrL23FC6AISsLXt7FAzSNByBEEIICYICQgghJAgKCCGEkCAoIIQQQoKggBBCCAmi+ZnoIvIGgKuly5GBWwG8WboQGZlz/eZcN4D1a5l+3XaVUrfFHKx5AZkrInIQu8xAzcy5fnOuG8D6tUzqutGFRQghJAgKCCGEkCAoIPXyWOkCZGbO9Ztz3QDWr2WS1o0xEEIIIUFwBEIIISQICgghhJAgKCAFEJGPichzInJdRE4P/ndeRK6IyGURuae3/y4Rebb73+dFRLr9N4rI17r9vy8iJyeujhER+YyI/JGIfKfb7u/9z6uuLSAi93b1uSIi50qXJwQReam7/t8RkYNu3y0i8k0Reb77+Z7e50fvYy2IyOMi8rqIXOrt865Pre1SU79pnjulFLeJNwB/G8DfAvA7AE739t8J4BkANwI4BeAFABvd/54CcDcAAfANAPd1+z8F4N91vz8I4Gul6zeo62cAfHpkv3dda98AbHT1uAPAVle/O0uXK6AeLwG4dbDvcwDOdb+fA/BZ232sZQPwEQAfAnAppj61tktN/SZ57jgCKYBS6v8qpS6P/OujAL6qlPpzpdSLAK4A+FkROQ7gryql/pda3un/AOAf9b7z5e73/wzgTC09Iwshda2dnwVwRSn1faXUTwB8Fct6zoF+O/syDre/I/dx+uLpUUr9LoAfDXZ71afmdqmpn46k9aOA1MXtAF7p/f1qt+/27vfh/kPfUUr9JYA/BbCTvaR+/FMR+W431F65CkLqWju6OrWGAvA/RORpETnb7XufUuo1AOh+vrfb32qdfevTYrvM/txRQDIhIt8SkUsjm6lHOjZyUIb9pu9MhqWuvwbgbwD4OwBeA/CvVl8bOZStrrXTctn7/D2l1IcA3Afgl0XkI4bPzqXOK+bSLid57vhK20wopf5BwNdeBfAzvb/fD+AH3f73j+zvf+dVEbkBwF+D+3A2Ca51FZF/D+A3uz9D6lo7ujo1hVLqB93P10Xkv2DpkvqhiBxXSr3WuTte7z7eap1969NUu1RK/XD1e87njiOQungCwINdZtUpAB8A8FQ3xH5LRD7cxTc+AeDrve883P3+AIDf7nyYVdA9nCt+DsAqUySkrrXzBwA+ICKnRGQLy6SGJwqXyQsRuUlEfnr1O4B/iOU967ezh3G4/R25j9OWOgiv+rTWLid77kpnEKzj1t3QVwH8OYAfAniy978LWGZGXEYvCwLA6a4RvADgUby7isBfAfCfsAyGPQXgjtL1G9T1KwCeBfDdrvEeD61rCxuA+wH8YVf2C6XLE1D+O7DM0nkGwHOrOmAZV/s2gOe7n7fY7mMtG4DfwNKN8xfdc/fJkPrU2i419ZvkueNSJoQQQoKgC4sQQkgQFBBCCCFBUEAIIYQEQQEhhBASBAWEEEJIEBQQQgghQVBACCGEBPH/Ac/WVNwz6pe6AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pca_analysis(2,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As you can see, 0 and 1 can be clearly separated by a straight line. This indicates that in the original 784-dimensional space dots corresponding to digits are also linearly separable. In the case of 2 and 5, we cannot find the good projection that will separate the digits clearly, and thus there are some cases of wrong classification.\n",
    "\n",
    "> Later on this course we will learn how to create non-linear classifiers using Neural Networks, and how to deal with a problem of digits not being properly aligned. Very soon we will reach above 99% accuracy in MNIST digit classification, while classifying them into 10 different classes.\n",
    "\n",
    "## Takeaway\n",
    "\n",
    " * We have leart about the simplest neural network architecture - one-layer perceptron.\n",
    " * We have implemented the perceptron \"by hand\", using simple training procedure based on gradient descent\n",
    " * Despite simplicity, one-layered perceptron can solve rather complex problems of handwritten digit recognition\n",
    " * One-layered perceptron is a liner classifier, and thus it provides the same classification power as logistic regression.\n",
    " * In the sample space, perceptron can separate two classes of input data using hyperplane."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Credits\n",
    "\n",
    "This notebook is a part of [AI for Beginners Curricula](http://github.com/microsoft/ai-for-beginners), and has been prepared by [Dmitry Soshnikov](http://soshnikov.com). It is inspired by Neural Network Workshop at Microsoft Research Cambridge. Some code and illustrative materials are taken from presentations by [Katja Hoffmann](https://www.microsoft.com/en-us/research/people/kahofman/), [Matthew Johnson](https://www.microsoft.com/en-us/research/people/matjoh/) and [Ryoto Tomioka](https://www.microsoft.com/en-us/research/people/ryoto/), and from [NeuroWorkshop](http://github.com/shwars/NeuroWorkshop) repository."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "interpreter": {
   "hash": "16aeaa504b544176258e5caf576fc030dfd6fff62d0c15825e7863ff13e121ff"
  },
  "kernelspec": {
   "display_name": "Python 3.8.0 64-bit (conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "livereveal": {
   "start_slideshow_at": "selected"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}