03.forward_propagation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Neural Prediction: Forward Propagation\n",
    "\n",
    "In this chapter, we will:\n",
    "- Create a simple network that makes a prediction.\n",
    "- Understand what exactly is a neural network, and what does it do?\n",
    "- We will make a prediction with multiple inputs.\n",
    "- We will make a prediction with multiple outputs.\n",
    "- We will make a prediction with multiple inputs and outputs.\n",
    "- We will use predictions to make other predictions.\n",
    "\n",
    "> [Warren Ellis] I try not to get envolved in the business of prediction. It's a quick way to look like an idiot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Predict "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, we have learned about the paradigm \"Predict, Compare, Learn\". In this chapter, we'll dive deep into the first step: **Predict**. For our first neural network, we will be predicting one data point at a time (single input, single output), like so:\n",
    "\n",
    "<div style=\"text-align:center;\"><img style=\"width:400px;\" src=\"static/imgs/03/one-to-one.png\" /></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"shape\" of the input has a significant impact on what a network looks like (its architecture). \n",
    "\n",
    "We usually start by inputting all of the information that correspond to the input entity (example: all pixel values of a cat image). If we are short on computational resources, we limit the inputs to the ones that we think will be \"most\" helpful in prediction. \n",
    "\n",
    "As a result, we can create a network only after we understand the **shape** of the input and output data sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to build a network with a single knob mapping from the input point to the output. In representation learning, these. knobs are acutally called \"weights\".\n",
    "\n",
    "Here's our first network, with a single \"weight\" mapping from the input \"# toes\" to the output \"win?\":\n",
    "\n",
    "<div style=\"text-align:center;\"><img style=\"width:400px;\" src=\"static/imgs/03/first-network.png\" /></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Simple Neural Network Making a prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with the simplest neural network possible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# an empty network.\n",
    "weight = .1\n",
    "def neural_network(x, w):\n",
    "    prediction = x * w\n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8500000000000001\n"
     ]
    }
   ],
   "source": [
    "# inserting one input datapoint.\n",
    "number_of_toes = [8.5, 9.5, 10, 9]\n",
    "x0 = number_of_toes[0]\n",
    "pred = neural_network(x0, weight)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the previous example, we can say that a neural network is of one or more weights that we can multiply by the input data to make a prediction. The input data is a numerical value that we measure in the real world, and a prediction is what. the neural network tells us, given the input data.\n",
    "\n",
    "The prediction, however, is not always right. Sometimes the neural network makes mistakes, but what is important is that it learns from them. A neural network learns by following these steps:\n",
    "1. It tries to make a prediction\n",
    "2. It sees whether the predictin was too high or too low.\n",
    "3. It changes the weight (up or down) to predict mre accurately the next time it sees the same input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Internals\n",
    "### It multiplies the Input by a Weight. It \"Scales\" the Input by a Certain Amount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A neural network, in its simplest form, uses the power of multiplication. Some weight values make parts of the input bigger, while others make other parts smaller.\n",
    "\n",
    "A neural network considers the input value as \"given information\" and its weight values as **knowledge**. By using both, it outputs a prediction. A neural network uses the knowledge stored in its weights to interpret the information in the input. Weights can be interpreted as a measue of **sensitivity** between the input and the prediction (weight is a volume knob). \n",
    "\n",
    "As demonstrated in the previous example, the NN had access to only one input instance. This implies that if we were to feed in `number_of_toes[1]`, the NN wouldn't remmeber the prediction it made in the last timestep. All in all, a neural network knows only what you feed it as input, it forgets everything else.\n",
    "\n",
    "Later, we will learn how to give a neural network a \"short-term memory\" by feeding in multiple inputs at once."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a Prediction with Multiple Inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to use multiple inputs or \"features\" that describe the same input instance.\n",
    "\n",
    "\n",
    "<div style=\"text-align:center;\"><img style=\"width:400px;\" src=\"static/imgs/03/many-to-one.png\" /></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our implementation.\n",
    "def w_sum(W, X):\n",
    "    \"\"\"Calculates W*X\"\"\"\n",
    "    assert(len(W) == len(X))\n",
    "    muls = list()\n",
    "    for i in range(slen(ws)):\n",
    "        muls.append(W[i] * X[i])\n",
    "    return sum(muls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def w_sum(a,b):\n",
    "    \"\"\"Books implementation of W*X\"\"\"\n",
    "    assert(len(a) == len(b))\n",
    "    output = 0\n",
    "    for i in range(len(a)):\n",
    "        output += (a[i] * b[i])\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# An empty network with multiple inputs.\n",
    "weights = [.1, .2, 0]\n",
    "def neural_network(X, weights):\n",
    "    pred = w_sum(X, weights)\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset provides the following information at the beginning of each game for the first four games in a season:\n",
    "- *toes*: current average number of toes per player.\n",
    "- *wlrec*: current games won (in percent).\n",
    "- *nfans*: fan count (in millions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "toes = [8.5, 9.5, 9.9, 9]\n",
    "wlrec = [.65, .8, .8, .9]\n",
    "nfans = [1.2, 1.3, .5, 1]\n",
    "x0 = [toes[0], wlrec[0], nfans[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9800000000000001"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neural_network(x0, weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Inputs: What Does this Network Do?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network simply multiples the 3 inputs by its internal 3 knob weights then sums them. This is what we call a **weighted sum**.\n",
    "\n",
    "As a result of the input having 3 values, the network also has 3 knobs, we multiply each input by its own weight. Because we have multiple inputs, we have to sum their respective local predictions. This is called the weighted sum or the **dot product**.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Challenge: Vector Math\n",
    "Being able to manipulate vectors is a cornerstore technique for representation learning. Let's implement simple functions that support the following operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def elementwise_multiplication(vec_a, vec_b):\n",
    "    \"\"\"Element-wise multiplication of two vectors of the same size.\"\"\"\n",
    "    assert(len(vec_a) == len(vec_b))\n",
    "    muls = list()\n",
    "    for i in range(len(vec_a)):\n",
    "        muls.append(vec_a[i] * vec_b[i])\n",
    "    return muls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def elementwise_addition(vec_a, vec_b):\n",
    "    \"\"\"Element-wise addition of two vectors of the same size.\"\"\"\n",
    "    assert(len(vec_a) == len(vec_b))\n",
    "    adds = list()\n",
    "    for i in range(len(vec_b)):\n",
    "        adds.append(vec_a[i] + vec_b[i])\n",
    "    return adds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vector_sum(vec_a):\n",
    "    \"\"\"Sums the values of a vector.\"\"\"\n",
    "    assert(type(vec_a) == list)\n",
    "    return sum(vec_a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vector_avg(vec_a):\n",
    "    \"\"\"Averages the values in a vector.\"\"\"\n",
    "    assert(type(vec_a) == list)\n",
    "    return sum(vec_a) / len(vec_a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a, b = [1,2,3], [4,5,6]\n",
    "vector_sum(elementwise_multiplication(a, b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The intuition behind the dot product operation is one of the most important parts of truly understanding how neural networks make predictions. Because we are summing up element-wise multiplications, the dot product gives us a measure of **similarity between two vectors**. Let's consider the following example:\n",
    "\n",
    "```\n",
    "a = [ 0, 1, 0, 1]         w_sum(a,b) = 0\n",
    "b = [ 1, 0, 1, 0]         w_sum(b,c) = 1\n",
    "c = [ 0, 1, 1, 0]         w_sum(b,d) = 1\n",
    "d = [.5, 0,.5, 0]         w_sum(c,c) = 2\n",
    "e = [ 0, 1,-1, 0]         w_sum(c,e) = 0\n",
    "```\n",
    "\n",
    "We can equate the properties of the dot product to the logical `AND`. In this analogy, negative weights tend to imply a logical `NOT` operator, and as we can observe, positive weights pairs with negative weights will cause the overall similarity score to go down.\n",
    "\n",
    "Neural networks are also able to model partial `AND`ing. But after multiplication, comes the summation (`OR`). In the case of `OR`, if any feature result in a high product, it will effect the overall score.\n",
    "\n",
    "Amusingly, this gives us a kind of crude language for reading weights. The following examples assume we're performing the dot product. In the case when `if` statments return `True`, we give back a high score:\n",
    "\n",
    "```\n",
    "W = [ 1, 0, 1] => if input[0] OR input[2]\n",
    "W = [ 0, 0, 1] => if input[2]\n",
    "W = [ 1, 0,-1] => if input[0] OR NOT input[2]\n",
    "W = [-1, 0,-1] => if NOT input[0] OR NOT input[2]\n",
    "W = [.5, 0, 1] => if BIG input[0] or input[2]\n",
    "```\n",
    "\n",
    "Takeaways:\n",
    "- The neural network gives a high score to the input most similar to the weights.\n",
    "- `nfans` is completely ignored in the prediction since its corresponding weight is **0**.\n",
    "- Shuffling the weights completely changes the way the network makes predictions.\n",
    "\n",
    "This analogy will help us significantly in the future, especially when putting networks together in increasingly complex ways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Inputs: Complete Runnable Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now's the time to use `numpy` (numerical python), which is a Python library, to optimize our neural network implementation: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.array([.1, .2, 0])\n",
    "toes = [8.5, 9.5, 9.9, 9]\n",
    "wlrec = [.65, .8, .8, .9]\n",
    "nfans = [1.2, 1.3, .5, 1]\n",
    "\n",
    "def neural_network(W, X):\n",
    "    assert(W.shape[0] == X.shape[0])\n",
    "    return np.dot(weights, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9800000000000001"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neural_network(weights, np.array([toes[0], wlrec[0], nfans[0]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a Prediction with Multiple Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural networks can also make multiple predictions using only a single input. Prediction occurs the same as if there were 3 disconnected single-weight neural networks:\n",
    "\n",
    "<div style=\"text-align:center;\"><img style=\"width:400px;\" src=\"static/imgs/03/NN-1-to-many.png\" /></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neural_network(W, X):\n",
    "    \"\"\"Note: NumPy performs accurate dot products based on the Input & Weight shapes.\"\"\"\n",
    "    return X * W"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.6, 2.1, 0. ])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neural_network(np.array([3]), np.array([.2, .7, 0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = [.3, .2, .9]\n",
    "\n",
    "def neural_network(W, X):\n",
    "    \"\"\"Book's Implementation.\"\"\"\n",
    "    pred = ele_mul(X, W)\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ele_mul(c, l):\n",
    "    \"\"\"Element-wise scalar multiplication.\"\"\"\n",
    "    assert(type(c) == int)\n",
    "    result = list()\n",
    "    for i in range(len(l)):\n",
    "        result.append(c * l[i])\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should note that the three predictions are completely separate. Unike neural networks with multiple inputs & a single output where we sum products. This network truly behaves as three independent components, each receiving the same input data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting with Multiple Inputs & Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, neural networks can also predict multiple outputs given multiple inputs:\n",
    "\n",
    "<div style=\"text-align:center;\"><img style=\"width:200px;\" src=\"static/imgs/03/NN-many-to-many.png\" /></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We set the weights (knowledge) values\n",
    "         #toes #%win #fans\n",
    "weights = [[.1, .1, -.3],  # 1st Neuron: Hurt ?\n",
    "           [.1, .2, .0],   # win ?\n",
    "           [.0, 1.3, .1]]  # Sad ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neural_network(X, W):\n",
    "    pred = vect_mat_mult(X, W)\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input [R^{1x3}] ; Weights [R_{3x3}]\n",
    "def vect_mat_mult(vect, matrix):\n",
    "    \"\"\"Calculates X (vect) * W (matrix)\"\"\"\n",
    "    assert(len(vect) == len(matrix))\n",
    "    output = [0] * len(vect)\n",
    "    for i in range(len(vect)):\n",
    "        output[i] = w_sum(vect, matrix[i])\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs.\n",
    "toes = [8.5, 9.5, 9.9, 9.0]\n",
    "wlrec = [.65, .8, .8, .9]\n",
    "nfans = [1.2, 1.3, .5, 1.0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one column.\n",
    "x0 = [toes[0], wlrec[0], nfans[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.555, 0.9800000000000001, 0.9650000000000001]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = neural_network(x0, weights); pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Inputs & Outputs: How does it work ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of multiple inputs & outputs, the network performs three independent weighted sums of the input to make three predictions:\n",
    "\n",
    "<div style=\"text-align:center;\"><img style=\"width:300px;\" src=\"static/imgs/03/many-to-many-NN.png\" /></div>\n",
    "\n",
    "The network performs three independent dot products (weighted sums) to output the predictions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting on Predictions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural networks can also be **stacked**:\n",
    "\n",
    "<div style=\"text-align:center;\"><img style=\"width:300px;\" src=\"static/imgs/03/deep-NN.png\" /></div>\n",
    "\n",
    "We can also take the output of one network and feed it as input to another network. This will result in two consecutive vector-matrix multiplications. Let's try an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A Network with multiple inputs and outputs\n",
    "ih_wgt = [[0.1, 0.2, -0.1],\n",
    "          [-0.1,0.1, 0.9],\n",
    "          [0.1, 0.4, 0.1]]\n",
    "hp_wgt = [[0.3, 1.1, -0.3],\n",
    "          [0.1, 0.2, 0.0],\n",
    "          [0.0, 1.3, 0.1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = [ih_wgt, hp_wgt]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neural_network(X, W):\n",
    "    hid = vect_mat_mult(X, W[0])\n",
    "    pred = vect_mat_mult(hid, W[1])\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.21350000000000002, 0.14500000000000002, 0.5065]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = neural_network(x0, weights); pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example shows us how we can perform the same computatios using a convenient Python library called `Numpy`. Using libraries like `NumPy` makes our code faster and easier to read and write."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "ih_wgt = np.array(ih_wgt).transpose()\n",
    "hp_wgt = np.array(hp_wgt).transpose()\n",
    "weights = [ih_wgt, hp_wgt]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neural_network(X, W):\n",
    "    out = X.dot(W[0])\n",
    "    pred = out.dot(W[1])\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "toes = np.array(toes)\n",
    "wlrec = np.array(wlrec)\n",
    "nfans = np.array(nfans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "x0 = np.array([toes[0], wlrec[0], nfans[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.2135, 0.145 , 0.5065])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = neural_network(x0, weights); pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Quick Primer on NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Numpy` adds support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays. We note the following before diving into examples:\n",
    "\n",
    "- We will keep using native python functions to be sure we fully understand what is going on inside them.\n",
    "- We call a matrix with only one row as a vector.\n",
    "- We create matrices by listing (rows, columns): **Rows come first, Columns come Second**.s\n",
    "\n",
    "Let's check some `numpy` examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([0,1,2,3])  # a vector.\n",
    "b = np.array([4,5,6,7])  # another vector.\n",
    "c = np.array([[0,1,2,3], [4,5,6,7]])  # A Matrix.\n",
    "d = np.zeros((2,4))  # 2x4 matrix of zeros.\n",
    "e = np.random.rand(2,5)  # 2x5 matrix of random number between 0 & 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3]\n",
      "[4 5 6 7]\n",
      "[[0 1 2 3]\n",
      " [4 5 6 7]]\n",
      "[[0. 0. 0. 0.]\n",
      " [0. 0. 0. 0.]]\n",
      "[[0.1645878  0.29578112 0.18441271 0.14036276 0.07897252]\n",
      " [0.27901318 0.69096197 0.60646666 0.01898684 0.88302135]]\n"
     ]
    }
   ],
   "source": [
    "print(a,b,c,d,e, sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.  0.2 0.4 0.6]\n"
     ]
    }
   ],
   "source": [
    "# element-wise multiplication.\n",
    "print(a*.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.  0.1 0.2 0.3]\n",
      " [0.4 0.5 0.6 0.7]]\n"
     ]
    }
   ],
   "source": [
    "# element-wise multiplication.\n",
    "print(c*.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0  5 12 21]\n"
     ]
    }
   ],
   "source": [
    "# multiply two vectors (element wise).\n",
    "print(a*b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.  1.5 3.6 6.3]\n"
     ]
    }
   ],
   "source": [
    "# complex element-wise multiplications.\n",
    "print(a*b*.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0  1  4  9]\n",
      " [ 0  5 12 21]]\n"
     ]
    }
   ],
   "source": [
    "# element-wise row multiplications (because of compatible shapes).\n",
    "print(a*c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "operands could not be broadcast together with shapes (4,) (2,5) ",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-58-82387bc21b72>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# error in case of incompatible shapes.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m: operands could not be broadcast together with shapes (4,) (2,5) "
     ]
    }
   ],
   "source": [
    "# error in case of incompatible shapes.\n",
    "print(a*e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we multiply two variables using the `*` operator, `numpy` automatically detects what kind of variables we are working with and tries to figure out what kind of operation we want. When we use (`+`, `-`, `/`), either the two variables should have the same number of columns or one of them should have `1` in its `shape`.\n",
    "\n",
    "When we are performing an operation using `numpy`, we should always keep the shapes of the inputs in mind. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 3)\n"
     ]
    }
   ],
   "source": [
    "a = np.zeros((1,4))\n",
    "b = np.zeros((4,3))\n",
    "c = a.dot(b)\n",
    "print(c.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we use the `.dot` `numpy` operator, we should pay attention to the order because we dot the variables that are next to each other. \n",
    "\n",
    "Let's check more examples that demonstrate the concept of `shape`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 3)\n"
     ]
    }
   ],
   "source": [
    "a = np.zeros((2,4))\n",
    "b = np.zeros((4,3))\n",
    "c = a.dot(b)\n",
    "print(c.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 3)\n"
     ]
    }
   ],
   "source": [
    "e = np.zeros((2,1))\n",
    "f = np.zeros((1,3))\n",
    "g = e.dot(f)\n",
    "print(g.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4, 6)\n"
     ]
    }
   ],
   "source": [
    "h = np.zeros((5,4)).T\n",
    "i = np.zeros((5,6))\n",
    "j = h.dot(i)\n",
    "print(j.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "shapes (5,4) and (5,6) not aligned: 4 (dim 1) != 5 (dim 0)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-64-86b642518282>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mh\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mi\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m6\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mj\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mh\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: shapes (5,4) and (5,6) not aligned: 4 (dim 1) != 5 (dim 0)"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "h = np.zeros((5,4))\n",
    "i = np.zeros((5,6))\n",
    "j = h.dot(i)\n",
    "print(j.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "### To Predict, Neural Networks Perform Repeated Weighted Sums of the Input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To predict, neural networks perform repeated weighted sums of the input. The network's \"intelligence\" depends on the weight values we give it. \n",
    "\n",
    "Everything we have done in this chapter is a form of what is called **forward propagation**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sketches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center;\"><img style=\"width:400px\" src=\"static/imgs/03/1-to-1-NN-Note.jpg\" /><img style=\"width:333px\" src=\"static/imgs/03/Reminder-1-to-1-NN-Note.jpg\" /></div>\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}