t81_558_class_05_1_reg_ridge_lasso.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_1_reg_ridge_lasso.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 5: Regularization and Dropout**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 5 Material\n",
    "\n",
    "* **Part 5.1: Part 5.1: Introduction to Regularization: Ridge and Lasso** [[Video]](https://www.youtube.com/watch?v=jfgRtCYjoBs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_1_reg_ridge_lasso.ipynb)\n",
    "* Part 5.2: Using K-Fold Cross Validation with Keras [[Video]](https://www.youtube.com/watch?v=maiQf8ray_s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_2_kfold.ipynb)\n",
    "* Part 5.3: Using L1 and L2 Regularization with Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=JEWzWv1fBFQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_3_keras_l1_l2.ipynb)\n",
    "* Part 5.4: Drop Out for Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=bRyOi0L6Rs8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_4_dropout.ipynb)\n",
    "* Part 5.5: Benchmarking Keras Deep Learning Regularization Techniques [[Video]](https://www.youtube.com/watch?v=1NLBwPumUAs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_5_bootstrap.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5.1: Introduction to Regularization: Ridge and Lasso\n",
    "\n",
    "Regularization is a technique that reduces overfitting, which occurs when neural networks attempt to memorize training data rather than learn from it. Humans are capable of overfitting as well. Before examining how a machine accidentally overfits, we will first explore how humans can suffer from it.\n",
    "\n",
    "Human programmers often take certification exams to show their competence in a given programming language. To help prepare for these exams, the test makers often make practice exams available. Consider a programmer who enters a loop of taking the practice exam, studying more, and then retaking the practice exam. The programmer has memorized much of the practice exam at some point rather than learning the techniques necessary to figure out the individual questions. The programmer has now overfitted for the practice exam. When this programmer takes the real exam, his actual score will likely be lower than what he earned on the practice exam.\n",
    "\n",
    "Although a neural network received a high score on its training data, this result does not mean that the same neural network will score high on data that was not inside the training set. A computer can overfit as well. Regularization is one of the techniques that can prevent overfitting. Several different regularization techniques exist. Most work by analyzing and potentially modifying the weights of a neural network as it trains.  \n",
    "\n",
    "## L1 and L2 Regularization\n",
    "\n",
    "L1 and L2 regularization are two standard regularization techniques that can reduce the effects of overfitting. These algorithms can either work with an objective function or as part of the backpropagation algorithm. The regularization algorithm is attached to the training algorithm by adding an objective in both cases.  \n",
    "\n",
    "These algorithms work by adding a weight penalty to the neural network training. This penalty encourages the neural network to keep the weights to small values. Both L1 and L2 calculate this penalty differently. You can add this penalty calculation to the calculated gradients for gradient-descent-based algorithms, such as backpropagation. The penalty is negatively combined with the objective score for objective-function-based training, such as simulated annealing.\n",
    "\n",
    "We will look at linear regression to see how L1 and L2 regularization work. The following code sets up the auto-mpg data for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LassoCV\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "names = ['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']\n",
    "x = df[names].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=45)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the data just loaded for several examples. The first examples in this part use several forms of linear regression. For linear regression, it is helpful to examine the model's coefficients. The following function is utilized to display these coefficients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple function to evaluate the coefficients of a regression\n",
    "%matplotlib inline    \n",
    "from IPython.display import display, HTML    \n",
    "\n",
    "def report_coef(names,coef,intercept):\n",
    "    r = pd.DataFrame( { 'coef': coef, 'positive': coef>=0  }, index = names )\n",
    "    r = r.sort_values(by=['coef'])\n",
    "    display(r)\n",
    "    print(f\"Intercept: {intercept}\")\n",
    "    r['coef'].plot(kind='barh', color=r['positive'].map(\n",
    "        {True: 'b', False: 'r'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression\n",
    "\n",
    "Before jumping into L1/L2 regularization, we begin with linear regression.  Researchers first introduced the L1/L2 form of regularization for [linear regression](https://en.wikipedia.org/wiki/Linear_regression).  We can also make use of L1/L2 for neural networks.  To fully understand L1/L2 we will begin with how we can use them with linear regression.\n",
    "\n",
    "The following code uses linear regression to fit the auto-mpg data set.  The RMSE reported will not be as good as a neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 3.0019345985860784\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.427721</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007255</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.005491</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.020166</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.138575</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.783047</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <td>1.003762</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "cylinders    -0.427721     False\n",
       "weight       -0.007255     False\n",
       "horsepower   -0.005491     False\n",
       "displacement  0.020166      True\n",
       "acceleration  0.138575      True\n",
       "year          0.783047      True\n",
       "origin        1.003762      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -19.101231042200112\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAD4CAYAAABIQCkOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZIElEQVR4nO3deZhkdX3v8fdHQBEHWWRAQsBRBAliBKZEUFkUQogaxYACLmELc9HE9UIefVyuG66Jxrghcg3ojWhQQK4xsskwiAzSA7MACihL1HC1CYSIhkX43j/qtCna7unume6uU93v1/P006fO+Z3f+daZnv7071enTqWqkCSpTR7V7wIkSRrNcJIktY7hJElqHcNJktQ6hpMkqXU27HcBc8VWW21VixYt6ncZkjQwVqxYcWdVLRxrm+E0TRYtWsTQ0FC/y5CkgZHk9vG2Oa0nSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJreOl5JpXkn5XIM0tM/XBFo6cJEmtYzhJklrHcJIktY7hJElqnTkfTkm+lWTzCdq8N8lBs1WTJGnt5uzVekkCpKpeOFHbqnrXLJQkSZqkgR45JXlLkuuarzclWZTkB0k+A1wDbJ/ktiRbNe3fmeSHSS5KclaSk5r1ZyQ5vFm+Lcl7klyTZE2SXfr3DCVpfhrYcEqyGDgWeDawN3ACsAXwNOCLVbVHVd3e074DHAbsAfwZ0FlL93dW1Z7AZ4GT1lLDkiRDSYaGh4fX9ylJkhoDG07A84Bzq+pXVXUvcA6wL3B7VS0fp/03quq/quqXwP9dS9/nNN9XAIvGa1RVp1VVp6o6CxeO+XlZkqR1MMjhNN57/X81xfZjub/5/hBz+HU5SWqrQQ6nZcChSTZJ8jjgZcDla2n/XeBPk2ycZAHwotkoUpI0dQM7Kqiqa5KcAXy/WXU6cPda2l+d5HxgFXA7MATcM9N1SpKmLjVTd+1roSQLqureJJvQHXktqaprpqPvTqdTQ0ND09GVZpA3fpWm1/pESJIVVTXmxWkDO3JaR6cl2RXYGDhzuoJJkjS95lU4VdUr+12DJGli8yqcpHk0iy0NtEG+Wk+SNEcZTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodPzJD84qfhKs28qNcfpcjJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juE0SUk26HcNkjRfzMlwSvK+JG/seXxKkjckOTnJ1UlWJ3lPz/bzkqxIcn2SJT3r703y3iRXAfvM8tOQpHlrToYT8L+BowGSPAo4Evg5sBOwF7A7sDjJfk3746pqMdAB3pDkCc36xwHXVdWzq+q7ow+SZEmSoSRDw8PDM/uMJGkemZPhVFW3Af+eZA/gYOBa4Fk9y9cAu9ANK+gG0ipgObB9z/qHgK+v5TinVVWnqjoLFy6ciaciSfPSXL5DxOnAMcATgS8ABwIfrKrP9TZKcgBwELBPVf06yVJg42bzfVX10GwVLEnqmpMjp8a5wCF0R0wXNF/HJVkAkGS7JFsDmwF3N8G0C7B3vwqWJHXN2ZFTVT2Q5FLgP5rRz4VJ/gC4Mt0brN0LvBr4NnBiktXAjXSn9iRJfTRnw6m5EGJv4OUj66rqE8Anxmj+J2P1UVULZqY6SdLazMlpvSS7Aj8CLqmqm/tdjyRpaubkyKmqbgCe0u86JEnrZk6GkzQePzdHGgxzclpPkjTYDCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1jh+ZoXklmf1j+jEd0tQ5cpIktY7hJElqHcNJktQ6hpMkqXVaGU5JlibpTFNfhybZtefxe5McNB19S5JmRivDaaqSbLCWzYcCvw2nqnpXVV0881VJktbVeoVTkvOSrEhyfZIlzbpDklyTZFWSS5p1C5L8Q5I1SVYnOaxZf3CSK5v2ZydZMMYxxmyT5LYk70ryXeDlSU5IcnVz3K8n2STJc4CXAB9NsjLJjknOSHJ408eBSa5t6vpCksf09P2e5phrkuyyPudJkjQ16ztyOq6qFgMd4A1JtgE+DxxWVc8EXt60eydwT1U9o6r+EPhOkq2AdwAHVdWewBDwlt7OJ9Hmvqp6XlV9BTinqp7VHPcHwPFV9T3gfODkqtq9qn7c0/fGwBnAEVX1DLrv+XptT993Nsf8LHDSWE8+yZIkQ0mGhoeHp3bmJEnjWt9wekOSVcByYHtgCbCsqm4FqKq7mnYHAZ8e2amq7gb2pjvddkWSlcDRwJNG9T9Rm6/2LO+W5PIka4BXAU+foPanAbdW1U3N4zOB/Xq2n9N8XwEsGquDqjqtqjpV1Vm4cOEEh5MkTdY63yEiyQF0Q2efqvp1kqXAKrq/9H+nOTD6ffIBLqqqo9Z2mAna/Kpn+Qzg0KpaleQY4ICJnsIE2+9vvj+Ed9KQpFm1PiOnzYC7m2Dahe4o5zHA/kmeDJBky6bthcBfjeyYZAu6o63nJnlqs26TJDuPOsZk2ozYFLgjyUZ0R04jftlsG+2HwKKRvoHXAJdN4nlLkmbY+oTTt4ENk6wG3kc3SIbpTu2d00z3jUy7vR/YIsl1zfrnV9UwcAxwVtPHcuARFx5Mpk2PdwJXARfRDZ4RXwFObi582LGn7/uAY4Gzm6nAh4FT1+VESJKmV8q7Uk6LTqdTQ0ND/S5DE/DGr1J7JFlRVWO+p3VOvM9JkjS3GE6SpNbxKjTNK06xSYPBkZMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWOH5mheWX0J+H6ERpSOzlykiS1juEkSWodw0mS1DqGkySpdaZ8QUSSdwP3Ao8HllXVxVPc/wDgpKp68VSPPduSHArcVFU39LsWSZpP1nnkVFXvmmowDaBDgV37XYQkzTeTCqckb09yY5KLgac1685Icniz/KEkNyRZneRverafmuTyJDcl+Z2RUpK9knwvybXN95G+N0jyN0nWNH2+vlm/OMllSVYkuSDJts36pUk+nmRZkh8keVaSc5LcnOT9Pcd7dZLvJ1mZ5HNJNmjW35vklCSrkixPsk2S5wAvAT7atN9xPc6zJGkKJpzWS7IYOBLYo2l/DbCiZ/uWwMuAXaqqkmzes/siYH9gR+DSJE8d1f0Pgf2q6jdJDgI+ABwGLAGeDOzRbNsyyUbAJ4GXVtVwkiOAU4Djmr4eqKr9krwR+AawGLgL+HGSjwNbA0cAz62qB5N8BngV8EXgccDyqnp7ko8AJ1TV+5OcD3yzqr42zrlZ0tTKDjvsMNGplCRN0mRec9oXOLeqfg3Q/MLu9Z/AfcDpSf4Z+GbPtn+qqoeBm5PcAuwyat/NgDOT7AQUsFGz/iDg1Kr6DUBV3ZVkN2A34KJ030m5AXBHT18jda0Brq+qO5p6bwG2B55HN7CubvZ/LPCLZp8HeupeAfzRJM4LVXUacBpAp9Px7ZySNE0me0HEuL94m5HNXsCBdEdYfwW8YJz9Rj9+H3BpVb0sySJgabM+Y7QN3dDZZ5xS7m++P9yzPPJ4w2b/M6vqbWPs+2DVb+8V8BDeOUOS+moyrzktA16W5LFJNgX+tHdjkgXAZlX1LeBNwO49m1+e5FHN6zVPAW4c1fdmwM+a5WN61l8InJhkw+YYWzb7LkyyT7NuoyRPn0T9Iy4BDk+y9UifSZ40wT6/BDadwjEkSdNgwnCqqmuArwIrga8Dl49qsinwzSSrgcuAN/dsu7FZ9y/AiVV136h9PwJ8MMkVdKfpRpwO/CuwOskq4JVV9QBwOPDhZt1K4DmTepbd53ED8A7gwqbWi4BtJ9jtK8DJzQUbXhAhSbMkNUN3vkxyBmu5mGCu6XQ6NTQ01O8yNAFv/Cq1R5IVVdUZa5t3iJAktc6MvfBfVcfMVN+SpLnNq9I0rziNJw0Gp/UkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCfNH6M/BldSaxlOkqTWMZwkSa1jOEmSWsdwkiS1zrSEU5JFSa6bjr4kSer7yCnJhv2uYTIGpU5JmgumM5w2SPL5JNcnuTDJY5PsnmR5ktVJzk2yBUCSpUk+kOQy4I1JXp7kuiSrkixr2myQ5KNJrm72/x/N+gOSLGv6uyHJqUke1Ww7Ksmapq8PN+tekeRjzfIbk9zSLO+Y5LvN8uIklyVZkeSCJNuOVec0nitJ0lpM52hgJ+CoqjohyT8BhwF/Dby+qi5L8l7gfwFvatpvXlX7AyRZA/xxVf0syebN9uOBe6rqWUkeA1yR5MJm217ArsDtwLeBP0vyPeDDwGLgbuDCJIcCy4CTm/32Bf49yXbA84DLk2wEfBJ4aVUNJzkCOAU4bnSdoyVZAiwB2GGHHdb1vEmSRpnOcLq1qlY2yyuAHen+Yr+sWXcmcHZP+6/2LF8BnNGE2jnNuoOBP0xyePN4M7oB+ADw/aoaGQGdRTdoHgSWVtVws/4fgf2q6rwkC5JsCmwPfBnYj25QnQM8DdgNuCjdN2luANwxTp2PUFWnAacBdDqdWvvpkSRN1nSG0/09yw8Bm4/XsPGrkYWqOjHJs4EXASuT7A6E7qjrgt6dkhwAjA6CatqP50rgWOBG4HK6o6J9gP8J7ABcX1X7TFSnJGl2zOQFEfcAdyfZt3n8GuCysRom2bGqrqqqdwF30h3hXAC8tpl2I8nOSR7X7LJXkic3rzUdAXwXuArYP8lWSTYAjuo53jLgpOb7tcDzgfur6h66gbUwyT7NcTZK8vTpOw2SpKma6SvQjgZOTbIJcAvd0ctYPppkJ7qjn0uAVcBqYBFwTbrzbcPAoU37K4EPAc+gGzjnVtXDSd4GXNr0862q+kbT/nK6gbesqh5K8hPghwBV9UAzdfj3STaje07+Drh+ms6BJGmKUjVYL5U003onVdWL+11Lr06nU0NDQ/0uQ2uTwID9vEtzWZIVVdUZa1vf3+ckSdJoA/fG0qpaCiztcxmSpBnkyEnzh1N60sAwnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ80PSfdL0kAwnCRJrWM4SZJax3CSJLWO4SRJap05HU5JTk+y6wRtzkhy+BjrFyV55cxVJ0kaz5wOp6r6i6q6YR13XwQYTpLUBwMRTkn+OskbmuWPJ/lOs3xgkv+T5OAkVya5JsnZSRY025cm6TTLxye5qVn3+SSf6jnEfkm+l+SWnlHUh4B9k6xM8uZZfLqSNO8NRDgBy4B9m+UOsCDJRsDzgDXAO4CDqmpPYAh4S+/OSX4PeCewN/BHwC6j+t+26evFdEMJ4K3A5VW1e1V9fKyikixJMpRkaHh4eD2foiRpxKCE0wpgcZJNgfuBK+mG1L7AfwG7AlckWQkcDTxp1P57AZdV1V1V9SBw9qjt51XVw80U4DaTLaqqTquqTlV1Fi5cuE5PTJL0uzbsdwGTUVUPJrkNOBb4HrAaeD6wI3ArcFFVHbWWLia6NcD9U2grSZphgzJygu7U3knN98uBE4GVwHLguUmeCpBkkyQ7j9r3+8D+SbZIsiFw2CSO90tg0+kqXpI0eYMUTpfTfW3oyqr6OXAf3deEhoFjgLOSrKYbVo94TamqfgZ8ALgKuBi4AbhnguOtBn6TZJUXREjS7EpV9buGWZFkQVXd24yczgW+UFXnTlf/nU6nhoaGpqs7TbeRm77Ok593aRAkWVFVnbG2DdLIaX29u7lg4jq6r1Od1+d6JEnjGIgLIqZDVZ3U7xokSZMzb8JJ85zTedJAmU/TepKkAWE4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6/h5Tm0w8hHimnl+rpM0EBw5SZJax3CSJLWO4SRJap2BCKckZyQ5vFk+PcmuU9z/3pmpTJI0Ewbugoiq+ouZ7D9JgFTVwzN5HEnS+Po6ckry50lWJ1mV5NwktybZqNn2+CS3jTzu2Wdpkk6zfG+SU5r9lyfZpln/5CRXJrk6yftG7X9ys351kvc06xYl+UGSzwDXANs3o7XrkqxJ8ubZOB+SpK6+hVOSpwNvB15QVc8EjgeWAi9qmhwJfL2qHlxLN48Dljf7LwNOaNZ/AvhsVT0L+H89xzwY2AnYC9gdWJxkv2bz04AvVtUewFbAdlW1W1U9A/iHcZ7DkiRDSYaGh4endgIkSePq58jpBcDXqupOgKq6CzgdOLbZfizjhEKPB4BvNssrgEXN8nOBs5rlL/W0P7j5upbuCGkXumEFcHtVLW+WbwGekuSTSQ4B/nOsg1fVaVXVqarOwoULJyhVkjRZ/XzNKcAj3hFZVVc0U2z7AxtU1XUT9PFg1W/fVfkQj3w+Y73bMsAHq+pzj1iZLAJ+1VPH3UmeCfwx8JfAK4DjJnxGkqRp0c+R0yXAK5I8ASDJls36L9Id9Uw0alqbK+hOCwK8qmf9BcBxSRY0x9wuydajd06yFfCoqvo68E5gz/WoRZI0RX0Lp6q6HjgFuCzJKuBjzaZ/BLbgv6fl1sUbgb9McjWwWc8xLwS+DFyZZA3wNWDTMfbfDliaZCVwBvC29ahFkjRFqZbda6x5P9NLq+o1/a5lKjqdTg0NDa3bzt5bb/a07Oddms+SrKiqzljbWvU+pySfBP4EeGG/a5Ek9U+rwqmqXt/vGiRJ/deqcJq3nGqSpEcYiHvrSZLmF8NJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWqd1ty8aVEmGgdunudutgDunuc+ZMCh1wuDUOih1wuDUOih1wuDUur51Pqmqxvy8IcOpxZIMjXffqTYZlDphcGodlDphcGodlDphcGqdyTqd1pMktY7hJElqHcOp3U7rdwGTNCh1wuDUOih1wuDUOih1wuDUOmN1+pqTJKl1HDlJklrHcJIktY7h1CJJtkxyUZKbm+9brKXt45P8LMmnZrPG5tgT1plk9yRXJrk+yeokR8xifYckuTHJj5K8dYztj0ny1Wb7VUkWzVZtY9QyUa1vSXJDcw4vSfKkftTZ1LLWWnvaHZ6kkvTlUujJ1JnkFc15vT7Jl2e7xp46Jvr33yHJpUmubX4G+vIp4Um+kOQXSa4bZ3uS/H3zPFYn2XO9D1pVfrXkC/gI8NZm+a3Ah9fS9hPAl4FPtbFOYGdgp2b594A7gM1nobYNgB8DTwEeDawCdh3V5nXAqc3ykcBX+/TvPZlanw9s0iy/ts21Nu02BZYBy4FOG+sEdgKuBbZoHm/d1nNK94KD1zbLuwK39anW/YA9gevG2f5C4F+AAHsDV63vMR05tctLgTOb5TOBQ8dqlGQxsA1w4SzVNdqEdVbVTVV1c7P8b8AvgDHfCT7N9gJ+VFW3VNUDwFeaenv11v814MAkmYXaRpuw1qq6tKp+3TxcDvz+LNc4YjLnFeB9dP94uW82i+sxmTpPAD5dVXcDVNUvZrnGEZOptYDHN8ubAf82i/X9dxFVy4C71tLkpcAXq2s5sHmSbdfnmIZTu2xTVXcANN+3Ht0gyaOAvwVOnuXaek1YZ68ke9H9y/DHs1DbdsBPeh7/tFk3Zpuq+g1wD/CEWahttMnU2ut4un+d9sOEtSbZA9i+qr45m4WNMplzujOwc5IrkixPcsisVfdIk6n13cCrk/wU+Bbw+tkpbcqm+rM8oQ3XqxxNWZKLgSeOsentk+zidcC3quonM/nH/jTUOdLPtsCXgKOr6uHpqG2iQ46xbvT7JSbTZjZMuo4krwY6wP4zWtH41lpr80fTx4FjZqugcUzmnG5Id2rvALoj0cuT7FZV/zHDtY02mVqPAs6oqr9Nsg/wpabW2fi/NBXT/n/KcJplVXXQeNuS/DzJtlV1R/NLfazphn2AfZO8DlgAPDrJvVU17gvUfaqTJI8H/hl4RzPUnw0/Bbbvefz7/O5UyEibnybZkO50ydqmLGbKZGolyUF0/yjYv6run6XaRpuo1k2B3YClzR9NTwTOT/KSqhqatSon/++/vKoeBG5NciPdsLp6dkp8RB0T1Xo8cAhAVV2ZZGO6N1vt11TkeCb1szwVTuu1y/nA0c3y0cA3RjeoqldV1Q5VtQg4ie4877QG0yRMWGeSRwPn0q3v7Fms7WpgpyRPbmo4km69vXrrPxz4TjWv6s6yCWttpso+B7ykj6+NwAS1VtU9VbVVVS1qfjaX0615NoNpwjob59G90IQkW9Gd5rtlVqvsmkyt/wocCJDkD4CNgeFZrXJyzgf+vLlqb2/gnpGp/3XWjys//Br3ipgnAJcANzfft2zWd4DTx2h/DP25Wm/COoFXAw8CK3u+dp+l+l4I3ET3Na63N+veS/eXJXT/g58N/Aj4PvCUPv6bT1TrxcDPe87h+W2tdVTbpfThar1JntMAHwNuANYAR7b1nNK9Qu8KulfyrQQO7lOdZ9G94vZBuqOk44ETgRN7zumnm+exZjr+7b19kSSpdZzWkyS1juEkSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrfP/AQ55MmH0t9qpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "\n",
    "# Create linear regression\n",
    "regressor = sklearn.linear_model.LinearRegression()\n",
    "\n",
    "# Fit/train linear regression\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(f\"Final score (RMSE): {score}\")\n",
    "\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L1 (Lasso) Regularization\n",
    "\n",
    "L1 regularization, also called LASSO (Least Absolute Shrinkage and Selection Operator) should be used to create sparsity in the neural network. In other words, the L1 algorithm will push many weight connections to near 0. When the weight is near 0, the program drops it from the network. Dropping weighted connections will create a sparse neural network.\n",
    "\n",
    "Feature selection is a useful byproduct of sparse neural networks. Features are the values that the training set provides to the input neurons. Once all the weights of an input neuron reach 0, the neural network training determines that the feature is unnecessary. If your data set has many unnecessary input features, L1 regularization can help the neural network detect and ignore unnecessary features.\n",
    "\n",
    "L1 is implemented by adding the following error to the objective to minimize:\n",
    "\n",
    "$$ E_1 = \\alpha \\sum_w{ |w| } $$\n",
    "\n",
    "You should use L1 regularization to create sparsity in the neural network. In other words, the L1 algorithm will push many weight connections to near 0. When the weight is near 0, the program drops it from the network. Dropping weighted connections will create a sparse neural network.\n",
    "\n",
    "The following code demonstrates lasso regression. Notice the effect of the coefficients compared to the previous section that used linear regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 3.0604021904033303\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.012995</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007328</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.002715</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.011601</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.114391</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <td>0.708222</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.777480</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "cylinders    -0.012995     False\n",
       "weight       -0.007328     False\n",
       "horsepower   -0.002715     False\n",
       "displacement  0.011601      True\n",
       "acceleration  0.114391      True\n",
       "origin        0.708222      True\n",
       "year          0.777480      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -18.506677982383252\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAD4CAYAAAC5S3KDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAYcElEQVR4nO3deZhldX3n8feHZVRoBJTWcRBsRYS0qCAFgsqiEkJcMaCCy8giPThGXAKZ+LiMG655JI5RsWVMYyaigqDGUdmkaUAaqYZeAAWVZaLjmEYICoZF+M4f9zRcylq7a/lV9fv1PPXUub/zO7/zvae661O/c889N1WFJEmt2WSmC5AkaTgGlCSpSQaUJKlJBpQkqUkGlCSpSZvNdAFzxXbbbVcLFiyY6TIkadZYsWLFrVU1f6T1BtQkWbBgAYODgzNdhiTNGkluGW29p/gkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElN8jLzRiUzXYEkjW0qPxDDGZQkqUkGlCSpSQaUJKlJBpQkqUkGlCSpSQbUOCXZdKZrkKSNyZwMqCQfSvK2vscnJzkhyUlJrkyyOskH+tZ/M8mKJNcmWdTXfmeSDya5Ath3mp+GJG3U5mRAAf8TeCNAkk2AI4BfAzsDewO7A3sm2b/rf0xV7QkMACckeWzXviVwTVU9p6ouHbqTJIuSDCYZXLt27dQ+I0nayMzJgKqqm4HfJNkDOBi4Gtirb/kqYFd6gQW9UFoFLAd26Gu/H/jGKPtZXFUDVTUwf/6In7klSVoPc/lOEqcBRwH/EfgS8CLgo1X1hf5OSQ4EDgL2rarfJ1kKPLJbfXdV3T9dBUuSHjInZ1Cdc4BD6M2czu2+jkkyDyDJ9kkeB2wN3N6F067APjNVsCTpIXN2BlVV9ya5CPi3bhZ0XpI/AS5P70Z3dwKvB74PHJ9kNXA9vdN8kqQZNmcDqrs4Yh/gVevaqurTwKeH6f7nw41RVfOmpjpJ0ljm5Cm+JAuBnwEXVtVPZ7oeSdLEzckZVFVdBzxlpuuQJK2/ORlQc8FUfsaKJM0Gc/IUnyRp9jOgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElN8uM2GtX7VHpJmri58nE9zqAkSU0yoCRJTTKgJElNMqAkSU2a8wGV5LtJthmjzweTHDRdNUmSxjZnr+JLEiBV9eKx+lbV+6ahJEnSBMzqGVSSdya5pvt6e5IFSX6c5HPAVcAOSW5Osl3X/71JfpLk/CRnJDmxa1+S5PBu+eYkH0hyVZI1SXaduWcoSRuvWRtQSfYEjgaeA+wDHAdsC+wCfLmq9qiqW/r6DwCHAXsAfwEMjDL8rVX1bODzwImj1LAoyWCSwbVr127oU5Ik9Zm1AQU8Hzinqu6qqjuBs4H9gFuqavkI/b9VVf9eVb8D/nmUsc/uvq8AFozUqaoWV9VAVQ3Mnz9/vZ6EJGl4szmgRrrXwl0T7D+ce7rv9zOHX6eTpJbN5oBaBhyaZIskWwKvBC4Zpf+lwMuSPDLJPOAl01GkJGn9zNrZQVVdlWQJ8KOu6TTg9lH6X5nk28Aq4BZgELhjquuUJK2f1Fy5q+A4JJlXVXcm2YLeDGxRVV01GWMPDAzU4ODgZAwFeLNYSetvtvxaT7Kiqka8YG3WzqDW0+IkC4FHAqdPVjhJkibfRhVQVfXama5BkjQ+G1VAzSazZYouSVNlNl/FJ0mawwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKT/LiNRs22T9T140EkTTZnUJKkJhlQkqQmGVCSpCYZUJKkJjUZUEmWJhmYpLEOTbKw7/EHkxw0GWNLkqZOkwE1UUk2HWX1ocCDAVVV76uqC6a+KknShtiggEryzSQrklybZFHXdkiSq5KsSnJh1zYvyT8kWZNkdZLDuvaDk1ze9T8zybxh9jFsnyQ3J3lfkkuBVyU5LsmV3X6/kWSLJM8FXg58MsnKJDslWZLk8G6MFyW5uqvrS0ke0Tf2B7p9rkmy64YcJ0nSxG3oDOqYqtoTGABOSPJ44IvAYVX1LOBVXb/3AndU1TOq6pnAD5JsB7wHOKiqng0MAu/sH3wcfe6uqudX1VeBs6tqr26/PwaOraofAt8GTqqq3avq531jPxJYArymqp5B7z1hb+4b+9Zun58HThzuySdZlGQwyeDatWsnduQkSaPa0IA6IckqYDmwA7AIWFZVNwFU1W1dv4OAz67bqKpuB/ahd+rtsiQrgTcCTxoy/lh9vta3vFuSS5KsAV4HPH2M2ncBbqqqG7rHpwP7960/u/u+Algw3ABVtbiqBqpqYP78+WPsTpI0Eet9J4kkB9ILnn2r6vdJlgKr6P3i/6PuwNB7DQQ4v6qOHG03Y/S5q295CXBoVa1KchRw4FhPYYz193Tf78c7bkjStNuQGdTWwO1dOO1Kb7bzCOCAJE8GSPKYru95wF+u2zDJtvRmXc9L8tSubYskTxuyj/H0WWcr4FdJNqc3g1rnd926oX4CLFg3NvAG4OJxPG9J0jTYkID6PrBZktXAh+iFyVp6p/nO7k79rTsF92Fg2yTXdO0vqKq1wFHAGd0Yy4GHXYwwnj593gtcAZxPL3zW+SpwUncxxE59Y98NHA2c2Z0WfAA4dX0OhCRp8qW8y+ekGBgYqMHBwUkbz5vFSprrkqyoqhHf8zon3gclSZp7DChJUpO8Oq1RnjKTtLFzBiVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkh+30aihn6jrx29I2tg4g5IkNcmAkiQ1yYCSJDXJgJIkNWnCF0kkeT9wJ/BoYFlVXTDB7Q8ETqyql05039MtyaHADVV13UzXIkkbm/WeQVXV+yYaTrPQocDCmS5CkjZG4wqoJO9Ocn2SC4BdurYlSQ7vlj+W5Lokq5P8bd/6U5NckuSGJH80Y0qyd5IfJrm6+75u7E2T/G2SNd2Yb+3a90xycZIVSc5N8oSufWmSU5IsS/LjJHslOTvJT5N8uG9/r0/yoyQrk3whyaZd+51JTk6yKsnyJI9P8lzg5cAnu/47bcBxliRN0Jin+JLsCRwB7NH1vwpY0bf+McArgV2rqpJs07f5AuAAYCfgoiRPHTL8T4D9q+oPSQ4CPgIcBiwCngzs0a17TJLNgc8Ar6iqtUleA5wMHNONdW9V7Z/kbcC3gD2B24CfJzkFeBzwGuB5VXVfks8BrwO+DGwJLK+qdyf5BHBcVX04ybeB71TVWSMcm0Vdrey4445jHUpJ0gSM5zWo/YBzqur3AN0v7X6/Be4GTkvyv4Hv9K37elU9APw0yY3ArkO23Ro4PcnOQAGbd+0HAadW1R8Aquq2JLsBuwHnp/cu1k2BX/WNta6uNcC1VfWrrt4bgR2A59MLrSu77R8F/Gu3zb19da8A/nQcx4WqWgwsBhgYGPCttJI0icZ7kcSIv3y7Gc7ewIvozbT+EnjhCNsNffwh4KKqemWSBcDSrj3D9A294Nl3hFLu6b4/0Le87vFm3fanV9W7htn2vqoH79VwP95hQ5Jm3Hheg1oGvDLJo5JsBbysf2WSecDWVfVd4O3A7n2rX5Vkk+71m6cA1w8Ze2vgl93yUX3t5wHHJ9ms28djum3nJ9m3a9s8ydPHUf86FwKHJ3ncujGTPGmMbX4HbDWBfUiSJsmYAVVVVwFfA1YC3wAuGdJlK+A7SVYDFwPv6Ft3fdf2PeD4qrp7yLafAD6a5DJ6p+zWOQ34P8DqJKuA11bVvcDhwMe7tpXAc8f1LHvP4zrgPcB5Xa3nA08YY7OvAid1F3F4kYQkTaPUFN2FNMkSRrnAYK4ZGBiowcHBSRvPm8VKmuuSrKiqgZHWeycJSVKTpuxigKo6aqrGliTNfV6t1ihP6Una2HmKT5LUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA6pFQz9OV5I2QgaUJKlJBpQkqUkGlCSpSQaUJKlJkxJQSRYkuWYyxpIkCRqYQSXZbKZrGI/ZUqckzRWTGVCbJvlikmuTnJfkUUl2T7I8yeok5yTZFiDJ0iQfSXIx8LYkr0pyTZJVSZZ1fTZN8skkV3bb/5eu/cAky7rxrktyapJNunVHJlnTjfXxru3VST7VLb8tyY3d8k5JLu2W90xycZIVSc5N8oTh6pzEYyVJGsNkzgp2Bo6squOSfB04DPhr4K1VdXGSDwL/HXh713+bqjoAIMka4M+q6pdJtunWHwvcUVV7JXkEcFmS87p1ewMLgVuA7wN/keSHwMeBPYHbgfOSHAosA07qttsP+E2S7YHnA5ck2Rz4DPCKqlqb5DXAycAxQ+scKskiYBHAjjvuuL7HTZI0jMkMqJuqamW3vALYid4v94u7ttOBM/v6f61v+TJgSRdsZ3dtBwPPTHJ493hreiF4L/Cjqlo3EzqDXtjcByytqrVd+z8B+1fVN5PMS7IVsAPwFWB/emF1NrALsBtwfnpvkN0U+NUIdT5MVS0GFgMMDAzU6IdHkjQRkxlQ9/Qt3w9sM1LHzl3rFqrq+CTPAV4CrEyyOxB6s69z+zdKciAwNAyq6z+Sy4GjgeuBS+jNjvYF/grYEbi2qvYdq05J0vSZyosk7gBuT7Jf9/gNwMXDdUyyU1VdUVXvA26lN9M5F3hzdwqOJE9LsmW3yd5Jnty99vQa4FLgCuCAJNsl2RQ4sm9/y4ATu+9XAy8A7qmqO+iF1vwk+3b72TzJ0yfvMEiS1sdUX5n2RuDUJFsAN9KbxQznk0l2pjcLuhBYBawGFgBXpXfubS1waNf/cuBjwDPohc45VfVAkncBF3XjfLeqvtX1v4Re6C2rqvuT/AvwE4Cqurc7jfg/kmxN75j8HXDtJB0DSdJ6SNXseumkO8V3YlW9dKZr6TcwMFCDg4OTM1gCs+znIkkTlWRFVQ2MtH7G3wclSdJwZt2bT6tqKbB0hsuQJE0xZ1At8vSeJBlQkqQ2GVCSpCYZUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmGVCSpCbNus+DmtOSh5b9yA1JGzlnUJKkJhlQkqQmGVCSpCYZUJKkJs3pgEpyWpKFY/RZkuTwYdoXJHnt1FUnSRrNnA6oqnpTVV23npsvAAwoSZohsyKgkvx1khO65VOS/KBbflGS/5Xk4CSXJ7kqyZlJ5nXrlyYZ6JaPTXJD1/bFJH/ft4v9k/wwyY19s6mPAfslWZnkHdP4dCVJzJKAApYB+3XLA8C8JJsDzwfWAO8BDqqqZwODwDv7N07yn4D3AvsAfwrsOmT8J3RjvZReMAH8DXBJVe1eVacMV1SSRUkGkwyuXbt2A5+iJKnfbAmoFcCeSbYC7gEupxdU+wH/DiwELkuyEngj8KQh2+8NXFxVt1XVfcCZQ9Z/s6oe6E4HPn68RVXV4qoaqKqB+fPnr9cTkyQNb1bcSaKq7ktyM3A08ENgNfACYCfgJuD8qjpylCEyyjrohd54+0qSpsFsmUFB7zTfid33S4DjgZXAcuB5SZ4KkGSLJE8bsu2PgAOSbJtkM+Cwcezvd8BWk1W8JGliZlNAXULvtaLLq+rXwN30XiNaCxwFnJFkNb3AethrTFX1S+AjwBXABcB1wB1j7G818Ickq7xIQpKmX2ojuSlpknlVdWc3gzoH+FJVnTNZ4w8MDNTg4OCGDeLNYiVtRJKsqKqBkdbPphnUhnp/dxHFNfRet/rmDNcjSRrFrLhIYjJU1YkzXYMkafw2moCaFTytJ0kP2phO8UmSZhEDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkPw+qBf0f9b6Onw0laSPnDEqS1CQDSpLUJANKktSkWRFQSZYkObxbPi3Jwgluf+fUVCZJmiqz7iKJqnrTVI6fJECq6oGp3I8kaXQzOoNK8p+TrE6yKsk5SW5Ksnm37tFJbl73uG+bpUkGuuU7k5zcbb88yeO79icnuTzJlUk+NGT7k7r21Uk+0LUtSPLjJJ8DrgJ26GZt1yRZk+Qd03E8JEkPmbGASvJ04N3AC6vqWcCxwFLgJV2XI4BvVNV9owyzJbC8234ZcFzX/mng81W1F/D/+vZ5MLAzsDewO7Bnkv271bsAX66qPYDtgO2rareqegbwDyM8h0VJBpMMrl27dmIHQJI0qpmcQb0QOKuqbgWoqtuA04Cju/VHM0Iw9LkX+E63vAJY0C0/DzijW/7Hvv4Hd19X05sp7UovsABuqarl3fKNwFOSfCbJIcBvh9t5VS2uqoGqGpg/f/4YpUqSJmImX4MK8LB3o1bVZd3ptgOATavqmjHGuK/qwXe03s/Dn89w73QN8NGq+sLDGpMFwF19ddye5FnAnwFvAV4NHDPmM5IkTZqZnEFdCLw6yWMBkjyma/8yvdnPWLOn0VxG7xQhwOv62s8Fjkkyr9vn9kkeN3TjJNsBm1TVN4D3As/egFokSethxgKqqq4FTgYuTrIK+FS36p+AbXnoFN36eBvwliRXAlv37fM84CvA5UnWAGcBWw2z/fbA0iQrgSXAuzagFknSekg1ds+37v1Or6iqN8x0LRMxMDBQg4OD67ex9+KTtBFKsqKqBkZa39T7oJJ8Bvhz4MUzXYskaWY1FVBV9daZrkGS1IamAmqj5ek8Sfojs+JefJKkjY8BJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJalJztzqarZKsBW6ZxCG3A26dxPEmS6t1Qbu1tVoXtFtbq3VBu7W1WheMXNuTqmrEzyoyoBqVZHC0e1TNlFbrgnZra7UuaLe2VuuCdmtrtS5Y/9o8xSdJapIBJUlqkgHVrsUzXcAIWq0L2q2t1bqg3dparQvara3VumA9a/M1KElSk5xBSZKaZEBJkppkQM2wJIckuT7Jz5L8zTDrH5Hka936K5IsaKSu/ZNcleQPSQ6fjprGWdc7k1yXZHWSC5M8qaHajk+yJsnKJJcmWdhKbX39Dk9SSablcuVxHLOjkqztjtnKJG+ajrrGU1vX59Xdv7drk3ylhbqSnNJ3vG5I8m/TUdc4a9sxyUVJru7+j47+6elV5dcMfQGbAj8HngL8B2AVsHBIn/8KnNotHwF8rZG6FgDPBL4MHN7Q8XoBsEW3/ObpOF4TqO3RfcsvB77fSm1dv62AZcByYKCFuoCjgL+fjuO0HrXtDFwNbNs9flwLdQ3p/1bgSw0ds8XAm7vlhcDNo43pDGpm7Q38rKpurKp7ga8CrxjS5xXA6d3yWcCLkmSm66qqm6tqNfDAFNcy0bouqqrfdw+XA09sqLbf9j3cEpiuK5TG8+8M4EPAJ4C7G6trJoyntuOAz1bV7QBV9a+N1NXvSOCMaagLxldbAY/ulrcG/u9oAxpQM2t74F/6Hv+iaxu2T1X9AbgDeGwDdc2EidZ1LPC9Ka3oIeOqLclbkvycXhCc0EptSfYAdqiq70xTTeOqq3NYdzrorCQ7TE9p46rtacDTklyWZHmSQxqpC4Du9PaTgR9MQ10wvtreD7w+yS+A79Kb4Y3IgJpZw82Ehv5VPZ4+k20m9jke464ryeuBAeCTU1pR3y6Hafuj2qrqs1W1E/DfgPdMeVU9o9aWZBPgFOCvpqmeB3c9TNvQY/bPwIKqeiZwAQ+dTZhq46ltM3qn+Q6kN1M5Lck2DdS1zhHAWVV1/xTW0288tR0JLKmqJwIvBv6x+/c3LANqZv0C6P+L8In88ZT3wT5JNqM3Lb6tgbpmwrjqSnIQ8G7g5VV1T0u19fkqcOiUVvSQsWrbCtgNWJrkZmAf4NvTcKHEmMesqn7T9zP8IrDnFNc07tq6Pt+qqvuq6ibgenqBNdN1rXME03d6D8ZX27HA1wGq6nLgkfRuJDu86XjxzK8RX1TcDLiR3jR83YuKTx/S5y08/CKJr7dQV1/fJUzfRRLjOV570HuhducGf5Y79y2/DBhspbYh/ZcyPRdJjOeYPaFv+ZXA8laOGXAIcHq3vB2901uPnem6un67ADfT3YyhoWP2PeCobvlP6AXYiDVOS+F+jfpDfTFwQ/dL9d1d2wfp/fUPvb8wzgR+BvwIeEojde1F7y+mu4DfANc2UtcFwK+Bld3Xtxv6WX4auLar66LRQmK6axvSd1oCapzH7KPdMVvVHbNdWzlm9E5pfQq4DlgDHNFCXd3j9wMfm65jNYFjthC4rPt5rgQOHm08b3UkSWqSr0FJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkpr0/wGc3+l42MRneQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "# Create linear regression\n",
    "regressor = Lasso(random_state=0,alpha=0.1)\n",
    "\n",
    "# Fit/train LASSO\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(f\"Final score (RMSE): {score}\")\n",
    "\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L2 (Ridge) Regularization\n",
    "\n",
    "You should use Tikhonov/Ridge/L2 regularization when you are less concerned about creating a space network and are more concerned about low weight values.  The lower weight values will typically lead to less overfitting. \n",
    "\n",
    "$$ E_2 = \\alpha \\sum_w{ w^2 } $$\n",
    "\n",
    "Like the L1 algorithm, the $\\alpha$ value determines how important the L2 objective is compared to the neural network’s error.  Typical L2 values are below 0.1 (10%).  The main calculation performed by L2 is the summing of the squares of all of the weights.  The algorithm will not sum bias values.\n",
    "\n",
    "You should use L2 regularization when you are less concerned about creating a space network and are more concerned about low weight values.  The lower weight values will typically lead to less overfitting.  Generally, L2 regularization will produce better overall performance than L1.  However, L1 might be useful in situations with many inputs, and you can prune some of the weaker inputs.\n",
    "\n",
    "The following code uses L2 with linear regression (Ridge regression):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): {score}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.421393</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007257</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.005385</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.020006</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.138470</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.782889</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <td>0.994621</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "cylinders    -0.421393     False\n",
       "weight       -0.007257     False\n",
       "horsepower   -0.005385     False\n",
       "displacement  0.020006      True\n",
       "acceleration  0.138470      True\n",
       "year          0.782889      True\n",
       "origin        0.994621      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -19.07980074425469\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAD4CAYAAABIQCkOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZB0lEQVR4nO3deZgldX3v8fdHQBEHWWRAQsBRBAliBOaIoLIohBA1igEFXMIW5qKJ64U8+rhcN1wTjXHDkWsGvRENCsg1RjYZBpFBemAWQAFliRquNoEQ0bAI3/vHqTaHtteZ7j7V3e/X8/Rz6lT96lffqjnTn/7VqVMnVYUkSW3yqH4XIEnScIaTJKl1DCdJUusYTpKk1jGcJEmts3G/C5grttlmm1q0aFG/y5CkWWPVqlV3VtXCkZYZTlNk0aJFDAwM9LsMSZo1ktw+2jJP60mSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLreCm55o2k3xVIc890fbGFIydJUusYTpKk1jGcJEmtYzhJklpnzodTkm8l2XKcNu9NcshM1SRJGtucvVovSYBU1QvHa1tV75qBkiRJEzSrR05J3pLkuubnTUkWJflBks8A1wA7JrktyTZN+3cm+WGSi5KcleSUZv6yJEc207cleU+Sa5KsS7Jb//ZQkuanWRtOSRYDxwPPBvYFTgK2Ap4GfLGq9qqq23vad4AjgL2APwM6Y3R/Z1XtDXwWOGWMGpYkGUgyMDg4uKG7JElqzNpwAp4HnFtVv6qqe4FzgP2B26tq5Sjtv1FV/1VVvwT+7xh9n9M8rgIWjdaoqpZWVaeqOgsXjvh9WZKk9TCbw2m0z/v/apLtR3J/8/gQc/h9OUlqq9kcTiuAw5NsluRxwMuAy8do/13gT5NsmmQB8KKZKFKSNHmzdlRQVdckWQZ8v5l1BnD3GO2vTnI+sAa4HRgA7pnuOiVJk5earrv2tVCSBVV1b5LN6I68llTVNVPRd6fTqYGBganoStPEG79KU29DIiTJqqoa8eK0WTtyWk9Lk+wObAqcOVXBJEmaWvMqnKrqlf2uQZI0vnkVTprf5tEZbGnWm81X60mS5ijDSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa3jV2Zo3vCbcNV2fq3Lf3PkJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwmKMlG/a5BkuaLORlOSd6X5I09z09L8oYkpya5OsnaJO/pWX5eklVJrk+ypGf+vUnem+QqYL8Z3g1JmrfmZDgB/xs4FiDJo4CjgZ8DuwD7AHsCi5Mc0LQ/oaoWAx3gDUme0Mx/HHBdVT27qr47fCNJliQZSDIwODg4vXskSfPInAynqroN+PckewGHAtcCz+qZvgbYjW5YQTeQ1gArgR175j8EfH2M7Sytqk5VdRYuXDgduyJJ89JcvkPEGcBxwBOBLwAHAx+sqs/1NkpyEHAIsF9V/TrJcmDTZvF9VfXQTBUsSeqakyOnxrnAYXRHTBc0PyckWQCQZIck2wJbAHc3wbQbsG+/CpYkdc3ZkVNVPZDkUuA/mtHPhUn+ALgy3Zus3Qu8Gvg2cHKStcCNdE/tSZL6aM6GU3MhxL7Ay4fmVdUngE+M0PxPRuqjqhZMT3WSpLHMydN6SXYHfgRcUlU397seSdLkzMmRU1XdADyl33VIktbPnAwnaSR+V440e8zJ03qSpNnNcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLr+JUZmjeS/mzXr+qQJs+RkySpdQwnSVLrGE6SpNYxnCRJrdPKcEqyPElnivo6PMnuPc/fm+SQqehbkjQ9WhlOk5VkozEWHw78Npyq6l1VdfH0VyVJWl8bFE5JzkuyKsn1SZY08w5Lck2SNUkuaeYtSPIPSdYlWZvkiGb+oUmubNqfnWTBCNsYsU2S25K8K8l3gZcnOSnJ1c12v55ksyTPAV4CfDTJ6iQ7J1mW5Mimj4OTXNvU9YUkj+np+z3NNtcl2W1DjpMkaXI2dOR0QlUtBjrAG5JsB3weOKKqngm8vGn3TuCeqnpGVf0h8J0k2wDvAA6pqr2BAeAtvZ1PoM19VfW8qvoKcE5VPavZ7g+AE6vqe8D5wKlVtWdV/bin702BZcBRVfUMup/5em1P33c22/wscMpIO59kSZKBJAODg4OTO3KSpFFtaDi9IckaYCWwI7AEWFFVtwJU1V1Nu0OATw+tVFV3A/vSPd12RZLVwLHAk4b1P16br/ZM75Hk8iTrgFcBTx+n9qcBt1bVTc3zM4EDepaf0zyuAhaN1EFVLa2qTlV1Fi5cOM7mJEkTtd53iEhyEN3Q2a+qfp1kObCG7i/932kODP+cfICLquqYsTYzTptf9UwvAw6vqjVJjgMOGm8Xxll+f/P4EN5JQ5Jm1IaMnLYA7m6CaTe6o5zHAAcmeTJAkq2bthcCfzW0YpKt6I62npvkqc28zZLsOmwbE2kzZHPgjiSb0B05Dflls2y4HwKLhvoGXgNcNoH9liRNsw0Jp28DGydZC7yPbpAM0j21d05zum/otNv7ga2SXNfMf35VDQLHAWc1fawEHnHhwUTa9HgncBVwEd3gGfIV4NTmwoede/q+DzgeOLs5FfgwcPr6HAhJ0tRKeVfKKdHpdGpgYKDfZWgM3vhVapckq6pqxM+0zonPOUmS5hbDSZLUOl6FpnnD02vS7OHISZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsevzNC8MdI34fo1GlI7OXKSJLWO4SRJah3DSZLUOoaTJKl1Jn1BRJJ3A/cCjwdWVNXFk1z/IOCUqnrxZLc905IcDtxUVTf0uxZJmk/We+RUVe+abDDNQocDu/e7CEmabyYUTknenuTGJBcDT2vmLUtyZDP9oSQ3JFmb5G96lp+e5PIkNyX5nZFSkn2SfC/Jtc3jUN8bJfmbJOuaPl/fzF+c5LIkq5JckGT7Zv7yJB9PsiLJD5I8K8k5SW5O8v6e7b06yfeTrE7yuSQbNfPvTXJakjVJVibZLslzgJcAH23a77wBx1mSNAnjntZLshg4GtiraX8NsKpn+dbAy4DdqqqSbNmz+iLgQGBn4NIkTx3W/Q+BA6rqN0kOAT4AHAEsAZ4M7NUs2zrJJsAngZdW1WCSo4DTgBOavh6oqgOSvBH4BrAYuAv4cZKPA9sCRwHPraoHk3wGeBXwReBxwMqqenuSjwAnVdX7k5wPfLOqvjbKsVnS1MpOO+003qGUJE3QRN5z2h84t6p+DdD8wu71n8B9wBlJ/hn4Zs+yf6qqh4Gbk9wC7DZs3S2AM5PsAhSwSTP/EOD0qvoNQFXdlWQPYA/gonQ/TbkRcEdPX0N1rQOur6o7mnpvAXYEnkc3sK5u1n8s8ItmnQd66l4F/NEEjgtVtRRYCtDpdPw4pyRNkYleEDHqL95mZLMPcDDdEdZfAS8YZb3hz98HXFpVL0uyCFjezM8IbUM3dPYbpZT7m8eHe6aHnm/crH9mVb1thHUfrPrtvQIewjtnSFJfTeQ9pxXAy5I8NsnmwJ/2LkyyANiiqr4FvAnYs2fxy5M8qnm/5inAjcP63gL4WTN9XM/8C4GTk2zcbGPrZt2FSfZr5m2S5OkTqH/IJcCRSbYd6jPJk8ZZ55fA5pPYhiRpCowbTlV1DfBVYDXwdeDyYU02B76ZZC1wGfDmnmU3NvP+BTi5qu4btu5HgA8muYLuabohZwD/CqxNsgZ4ZVU9ABwJfLiZtxp4zoT2srsfNwDvAC5sar0I2H6c1b4CnNpcsOEFEZI0Q1LTdOfLJMsY42KCuabT6dTAwEC/y9AYvPGr1C5JVlVVZ6Rl3iFCktQ60/bGf1UdN119S5LmNq9K07zhKTxp9vC0niSpdQwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hpPlhpK/BldRahpMkqXUMJ0lS6xhOkqTWMZwkSa0zJeGUZFGS66aiL0mS+j5ySrJxv2uYiNlSpyTNBVMZThsl+XyS65NcmOSxSfZMsjLJ2iTnJtkKIMnyJB9IchnwxiQvT3JdkjVJVjRtNkry0SRXN+v/j2b+QUlWNP3dkOT0JI9qlh2TZF3T14ebea9I8rFm+o1Jbmmmd07y3WZ6cZLLkqxKckGS7UeqcwqPlSRpDFM5GtgFOKaqTkryT8ARwF8Dr6+qy5K8F/hfwJua9ltW1YEASdYBf1xVP0uyZbP8ROCeqnpWkscAVyS5sFm2D7A7cDvwbeDPknwP+DCwGLgbuDDJ4cAK4NRmvf2Bf0+yA/A84PIkmwCfBF5aVYNJjgJOA04YXudwSZYASwB22mmn9T1ukqRhpjKcbq2q1c30KmBnur/YL2vmnQmc3dP+qz3TVwDLmlA7p5l3KPCHSY5snm9BNwAfAL5fVUMjoLPoBs2DwPKqGmzm/yNwQFWdl2RBks2BHYEvAwfQDapzgKcBewAXpftBzY2AO0ap8xGqaimwFKDT6dTYh0eSNFFTGU7390w/BGw5WsPGr4YmqurkJM8GXgSsTrInELqjrgt6V0pyEDA8CKppP5orgeOBG4HL6Y6K9gP+J7ATcH1V7TdenZKkmTGdF0TcA9ydZP/m+WuAy0ZqmGTnqrqqqt4F3El3hHMB8NrmtBtJdk3yuGaVfZI8uXmv6Sjgu8BVwIFJtkmyEXBMz/ZWAKc0j9cCzwfur6p76AbWwiT7NdvZJMnTp+4wSJIma7qvQDsWOD3JZsAtdEcvI/lokl3ojn4uAdYAa4FFwDXpnm8bBA5v2l8JfAh4Bt3AObeqHk7yNuDSpp9vVdU3mvaX0w28FVX1UJKfAD8EqKoHmlOHf59kC7rH5O+A66foGEiSJilVs+utkua03ilV9eJ+19Kr0+nUwMBAv8vQaBKYZa91aa5LsqqqOiMt6/vnnCRJGm7WfbC0qpYDy/tchiRpGjly0vzgKT1pVjGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnzQ9JvyuQNAmGkySpdQwnSVLrGE6SpNYxnCRJrTOnwynJGUl2H6fNsiRHjjB/UZJXTl91kqTRzOlwqqq/qKob1nP1RYDhJEl9MCvCKclfJ3lDM/3xJN9ppg9O8n+SHJrkyiTXJDk7yYJm+fIknWb6xCQ3NfM+n+RTPZs4IMn3ktzSM4r6ELB/ktVJ3jyDuytJ896sCCdgBbB/M90BFiTZBHgesA54B3BIVe0NDABv6V05ye8B7wT2Bf4I2G1Y/9s3fb2YbigBvBW4vKr2rKqPj1RUkiVJBpIMDA4ObuAuSpKGzJZwWgUsTrI5cD9wJd2Q2h/4L2B34Iokq4FjgScNW38f4LKququqHgTOHrb8vKp6uDkFuN1Ei6qqpVXVqarOwoUL12vHJEm/a+N+FzARVfVgktuA44HvAWuB5wM7A7cCF1XVMWN0Md7tAe6fRFtJ0jSbLSMn6J7aO6V5vBw4GVgNrASem+SpAEk2S7LrsHW/DxyYZKskGwNHTGB7vwQ2n6riJUkTN5vC6XK67w1dWVU/B+6j+57QIHAccFaStXTD6hHvKVXVz4APAFcBFwM3APeMs721wG+SrPGCCEmaWamqftcwI5IsqKp7m5HTucAXqurcqeq/0+nUwMDAVHWnqZbAPHmtS7NFklVV1Rlp2WwaOW2odzcXTFxH932q8/pcjyRpFLPigoipUFWn9LsGSdLEzKeRk+YzT+lJs4rhJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrWM4SZJaZ9582WCrJf2uYH7wO52kWcORkySpdQwnSVLrGE6SpNaZFeGUZFmSI5vpM5LsPsn1752eyiRJ02HWXRBRVX8xnf0nCZCqeng6tyNJGl1fR05J/jzJ2iRrkpyb5NYkmzTLHp/ktqHnPessT9Jppu9Nclqz/sok2zXzn5zkyiRXJ3nfsPVPbeavTfKeZt6iJD9I8hngGmDHZrR2XZJ1Sd48E8dDktTVt3BK8nTg7cALquqZwInAcuBFTZOjga9X1YNjdPM4YGWz/grgpGb+J4DPVtWzgP/Xs81DgV2AfYA9gcVJDmgWPw34YlXtBWwD7FBVe1TVM4B/GGUfliQZSDIwODg4uQMgSRpVP0dOLwC+VlV3AlTVXcAZwPHN8uMZJRR6PAB8s5leBSxqpp8LnNVMf6mn/aHNz7V0R0i70Q0rgNuramUzfQvwlCSfTHIY8J8jbbyqllZVp6o6CxcuHKdUSdJE9fM9pwCP+FRkVV3RnGI7ENioqq4bp48Hq377ycqHeOT+jPSJywAfrKrPPWJmsgj4VU8ddyd5JvDHwF8CrwBOGHePJElTop8jp0uAVyR5AkCSrZv5X6Q76hlv1DSWK+ieFgR4Vc/8C4ATkixotrlDkm2Hr5xkG+BRVfV14J3A3htQiyRpkvoWTlV1PXAacFmSNcDHmkX/CGzFf5+WWx9vBP4yydXAFj3bvBD4MnBlknXA14DNR1h/B2B5ktXAMuBtG1CLJGmSUi2731jzeaaXVtVr+l3LZHQ6nRoYGFi/lb233sxo2Wtdmu+SrKqqzkjLWvU5pySfBP4EeGG/a5Ek9U+rwqmqXt/vGiRJ/deqcJq3PN0kSY8wK+6tJ0maXwwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJap3W3b5otkoyCNw+Rd1tA9w5RX1NN2udHtY6Pax1eqxvrU+qqhG/b8hwaqEkA6Pdb6ptrHV6WOv0sNbpMR21elpPktQ6hpMkqXUMp3Za2u8CJsFap4e1Tg9rnR5TXqvvOUmSWseRkySpdQwnSVLrGE4tkGTrJBclubl53GqMto9P8rMkn5rJGnu2P26tSfZMcmWS65OsTXLUDNd4WJIbk/woyVtHWP6YJF9tll+VZNFM1jeslvFqfUuSG5rjeEmSJ/WjzqaWMWvtaXdkkkrSt8ugJ1Jrklc0x/b6JF+e6Rp76hjvNbBTkkuTXNu8Dvr2TeFJvpDkF0muG2V5kvx9sy9rk+y93hurKn/6/AN8BHhrM/1W4MNjtP0E8GXgU22tFdgV2KWZ/j3gDmDLGapvI+DHwFOARwNrgN2HtXkdcHozfTTw1T4dy4nU+nxgs2b6tW2utWm3ObACWAl02lorsAtwLbBV83zbFte6FHhtM707cFs/am22fwCwN3DdKMtfCPwLEGBf4Kr13ZYjp3Z4KXBmM30mcPhIjZIsBrYDLpyhukYybq1VdVNV3dxM/xvwC2DET4FPg32AH1XVLVX1APAVujX36t2HrwEHJ8kM1ddr3Fqr6tKq+nXzdCXw+zNc45CJHFeA99H9A+a+mSxumInUehLw6aq6G6CqfjHDNQ6ZSK0FPL6Z3gL4txms75GFVK0A7hqjyUuBL1bXSmDLJNuvz7YMp3bYrqruAGgetx3eIMmjgL8FTp3h2oYbt9ZeSfah+xfhj2egNoAdgJ/0PP9pM2/ENlX1G+Ae4AkzUt0odTRGqrXXiXT/Ku2HcWtNshewY1V9cyYLG8FEjuuuwK5JrkiyMslhM1bdI02k1ncDr07yU+BbwOtnprT1MtnX9Kg2npJyNK4kFwNPHGHR2yfYxeuAb1XVT6b7j/wpqHWon+2BLwHHVtXDU1HbRDY7wrzhn5eYSJuZMOE6krwa6AAHTmtFoxuz1uaPp48Dx81UQWOYyHHdmO6pvYPojkYvT7JHVf3HNNc23ERqPQZYVlV/m2Q/4EtNrTP1f2oypuz/luE0Q6rqkNGWJfl5ku2r6o7mF/pIpxj2A/ZP8jpgAfDoJPdW1ahvTPexVpI8Hvhn4B3N8H6m/BTYsef57/O7p0GG2vw0ycZ0T5WMdapiukykVpIcQvcPgwOr6v4Zqm248WrdHNgDWN788fRE4PwkL6mqgRmrsmuir4GVVfUgcGuSG+mG1dUzU+Ij6hiv1hOBwwCq6sokm9K90Wq/TkWOZUKv6YnwtF47nA8c20wfC3xjeIOqelVV7VRVi4BT6J7XnfJgmoBxa03yaOBcujWePYO1QfeXyy5JntzUcTTdmnv17sORwHeqeTd3ho1ba3Oq7HPAS/r4vgiMU2tV3VNV21TVouY1upJuzTMdTOPW2jiP7sUmJNmG7mm+W2a0yq6J1PqvwMEASf4A2BQYnNEqJ+584M+bq/b2Be4Zehtg0vp11Yc/j7jC5QnAJcDNzePWzfwOcMYI7Y+jf1frjVsr8GrgQWB1z8+eM1jjC4Gb6L7P9fZm3nvp/rKE7n/us4EfAd8HntLHf/vxar0Y+HnPcTy/rbUOa7ucPl2tN8HjGuBjwA3AOuDoFte6O3AF3Sv5VgOH9rHWs+heffsg3VHSicDJwMk9x/XTzb6s25DXgLcvkiS1jqf1JEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6/x/OVgyXZ8+pB4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "# Create linear regression\n",
    "regressor = Ridge(alpha=1)\n",
    "\n",
    "# Fit/train Ridge\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {score}\")\n",
    "\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ElasticNet Regularization\n",
    "\n",
    "The ElasticNet regression combines both L1 and L2.  Both penalties are applied.  The amount of L1 and L2 are governed by the parameters alpha and beta.\n",
    "\n",
    "$$ a * {\\rm L}1 + b * {\\rm L}2 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 3.0450899960775013\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.274010</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007303</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.003231</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.016194</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.132348</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.777482</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <td>0.782781</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "cylinders    -0.274010     False\n",
       "weight       -0.007303     False\n",
       "horsepower   -0.003231     False\n",
       "displacement  0.016194      True\n",
       "acceleration  0.132348      True\n",
       "year          0.777482      True\n",
       "origin        0.782781      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -18.389355690429767\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAD4CAYAAABIQCkOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAYRElEQVR4nO3deZhldX3n8feHhojYyCKNEgK2IkgQI9hXFJVFIYSoUQwo4DJsoUcncR3Io4/LuOGa0XGcKLaMaXTiEhSQMUY2aRqRRqqbXgAFlCXRMFoEQkTDInznj3vaXMrqrlvdVXVP3Xq/nqeee+45v/M739+tevrTv3PPPTdVhSRJbbLFoAuQJGksw0mS1DqGkySpdQwnSVLrGE6SpNbZctAFDIuddtqpFi5cOOgyJGnWWLly5Z1VtWC8bYbTFFm4cCEjIyODLkOSZo0kt29om6f1JEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXW8lFyzWjLoCqS5bbq+2MKZkySpdQwnSVLrGE6SpNYxnCRJrTP04ZTkW0m2n6DN+5IcPlM1SZI2bmiv1ksSIFX1oonaVtW7Z6AkSVKfZvXMKclbk1zX/Lw5ycIkP0jyaWAVsFuS25Ls1LR/V5IfJrk4yZeTnNasX5rkmGb5tiTvTbIqybokew9uhJI0N83acEqyCDgJeDbwHOBUYAfgqcAXqmr/qrq9p30HOBrYH/hToLOR7u+sqmcCnwFO20gNi5OMJBkZHR3d3CFJkhqzNpyA5wPnVdUvq+pe4FzgIOD2qlqxgfbfqKp/r6pfAP93I32f2zyuBBZuqFFVLamqTlV1FiwY9/uyJEmbYDaH04buDfDLSbYfz/3N40MM8ftyktRWszmclgNHJdkmyWOAlwNXbKT9d4E/SbJ1kvnAi2eiSEnS5M3aWUFVrUqyFPh+s+os4O6NtL8myQXAGuB2YAS4Z7rrlCRNXmq67trXQknmV9W9SbahO/NaXFWrpqLvTqdTIyMjU9GVJsEbv0qDtTkRkmRlVY17cdqsnTltoiVJ9gG2Bs6eqmCSJE2tORVOVfWqQdcgSZrYnAonDZ85dFZamlNm89V6kqQhZThJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdfzKDM1qfhOuNDjT+ZU1zpwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGU5+SzBt0DZI0VwxlOCV5f5I39Tw/I8kbk5ye5Joka5O8t2f7+UlWJrk+yeKe9fcmeV+Sq4EDZ3gYkjRnDWU4Af8bOAEgyRbAccDPgD2BA4D9gEVJDm7an1xVi4AO8MYkj2vWPwa4rqqeXVXfHXuQJIuTjCQZGR0dnd4RSdIcMpThVFW3Af+SZH/gCOBa4Fk9y6uAvemGFXQDaQ2wAtitZ/1DwNc3cpwlVdWpqs6CBQumYyiSNCcN8x0izgJOBJ4AfB44DPhQVX22t1GSQ4HDgQOr6ldJlgFbN5vvq6qHZqpgSVLXUM6cGucBR9KdMV3Y/JycZD5Akl2T7AxsB9zdBNPewHMGVbAkqWtoZ05V9UCSy4B/bWY/FyX5feCqdG/Idi/wGuDbwOuSrAVupHtqT5I0QEMbTs2FEM8BXrF+XVV9EvjkOM3/eLw+qmr+9FQnSdqYoTytl2Qf4EfApVV186DrkSRNzlDOnKrqBuDJg65DkrRphjKcNHdM5/fJSBqcoTytJ0ma3QwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY5fmaFZLRl0BePzqzykzePMSZLUOoaTJKl1DCdJUusYTpKk1mllOCVZlqQzRX0dlWSfnufvS3L4VPQtSZoerQynyUoybyObjwJ+E05V9e6qumT6q5IkbarNCqck5ydZmeT6JIubdUcmWZVkTZJLm3Xzk/xNknVJ1iY5ull/RJKrmvbnJJk/zjHGbZPktiTvTvJd4BVJTk1yTXPcryfZJslzgZcCH0uyOskeSZYmOabp47Ak1zZ1fT7Jo3r6fm9zzHVJ9t6c10mSNDmbO3M6uaoWAR3gjUkeD3wOOLqqngG8omn3LuCeqnp6Vf0B8J0kOwHvBA6vqmcCI8Bbezvvo819VfX8qvoKcG5VPas57g+AU6rqe8AFwOlVtV9V/bin762BpcCxVfV0up/5en1P33c2x/wMcNp4g0+yOMlIkpHR0dHJvXKSpA3a3HB6Y5I1wApgN2AxsLyqbgWoqruadocDf71+p6q6G3gO3dNtVyZZDZwAPHFM/xO1+WrP8r5JrkiyDng18LQJan8qcGtV3dQ8Pxs4uGf7uc3jSmDheB1U1ZKq6lRVZ8GCBRMcTpLUr02+Q0SSQ+mGzoFV9asky4A1dP/R/63mwNjPzAe4uKqO39hhJmjzy57lpcBRVbUmyYnAoRMNYYLt9zePD+GdNCRpRm3OzGk74O4mmPamO8t5FHBIkicBJNmxaXsR8Bfrd0yyA93Z1vOSPKVZt02SvcYco582620L3JFkK7ozp/V+0Wwb64fAwvV9A68FLu9j3JKkabY54fRtYMska4H30w2SUbqn9s5tTvetP+32AWCHJNc1619QVaPAicCXmz5WAI+48KCfNj3eBVwNXEw3eNb7CnB6c+HDHj193wecBJzTnAp8GDhzU14ISdLUSnmHyinR6XRqZGRk0GXMOd74VZq9kqysqnE/0zoUn3OSJA0Xw0mS1DpehaZZzdNn0nBy5iRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa3jV2ZoVhv7Tbh+hYY0HJw5SZJax3CSJLWO4SRJah3DSZLUOpO+ICLJe4B7gccCy6vqkknufyhwWlW9ZLLHnmlJjgJuqqobBl2LJM0lmzxzqqp3TzaYZqGjgH0GXYQkzTV9hVOSdyS5McklwFObdUuTHNMsfzjJDUnWJvmrnu1nJrkiyU1JfmumlOSAJN9Lcm3zuL7veUn+Ksm6ps83NOsXJbk8ycokFybZpVm/LMknkixP8oMkz0pybpKbk3yg53ivSfL9JKuTfDbJvGb9vUnOSLImyYokj0/yXOClwMea9ntsxussSZqECU/rJVkEHAfs37RfBazs2b4j8HJg76qqJNv37L4QOATYA7gsyVPGdP9D4OCq+nWSw4EPAkcDi4EnAfs323ZMshXwKeBlVTWa5FjgDODkpq8HqurgJG8CvgEsAu4CfpzkE8DOwLHA86rqwSSfBl4NfAF4DLCiqt6R5KPAqVX1gSQXAN+sqq9t4LVZ3NTK7rvvPtFLKUnqUz/vOR0EnFdVvwJo/sHu9W/AfcBZSf4e+GbPtr+rqoeBm5PcAuw9Zt/tgLOT7AkUsFWz/nDgzKr6NUBV3ZVkX2Bf4OJ0P3k5D7ijp6/1da0Drq+qO5p6bwF2A55PN7CuafZ/NPDzZp8HeupeCfxhH68LVbUEWALQ6XT8+KckTZF+L4jY4D+8zczmAOAwujOsvwBeuIH9xj5/P3BZVb08yUJgWbM+47QN3dA5cAOl3N88PtyzvP75ls3+Z1fV28fZ98Gq39xb4CG8c4YkDVQ/7zktB16e5NFJtgX+pHdjkvnAdlX1LeDNwH49m1+RZIvm/ZonAzeO6Xs74KfN8ok96y8CXpdky+YYOzb7LkhyYLNuqyRP66P+9S4Fjkmy8/o+kzxxgn1+AWw7iWNIkqbAhOFUVauArwKrga8DV4xpsi3wzSRrgcuBt/Rsu7FZ9w/A66rqvjH7fhT4UJIr6Z6mW+8s4B+BtUnWAK+qqgeAY4CPNOtWA8/ta5TdcdwAvBO4qKn1YmCXCXb7CnB6c8GGF0RI0gxJTdOdMpMsZSMXEwybTqdTIyMjgy5jzvHGr9LslWRlVXXG2+YdIiRJrTNtb/xX1YnT1bckabh5VZpmNU/jScPJ03qSpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpNmr7FfgytpaBhOkqTWMZwkSa1jOEmSWsdwkiS1zpSEU5KFSa6bir4kSRr4zCnJloOuoR+zpU5JGgZTGU7zknwuyfVJLkry6CT7JVmRZG2S85LsAJBkWZIPJrkceFOSVyS5LsmaJMubNvOSfCzJNc3+/7lZf2iS5U1/NyQ5M8kWzbbjk6xr+vpIs+6VST7eLL8pyS3N8h5JvtssL0pyeZKVSS5Msst4dU7hayVJ2oipnA3sCRxfVacm+TvgaOAvgTdU1eVJ3gf8N+DNTfvtq+oQgCTrgD+qqp8m2b7ZfgpwT1U9K8mjgCuTXNRsOwDYB7gd+Dbwp0m+B3wEWATcDVyU5ChgOXB6s99BwL8k2RV4PnBFkq2ATwEvq6rRJMcCZwAnj61zrCSLgcUAu++++6a+bpKkMaYynG6tqtXN8kpgD7r/sF/erDsbOKen/Vd7lq8Eljahdm6z7gjgD5Ic0zzfjm4APgB8v6rWz4C+TDdoHgSWVdVos/5vgYOr6vwk85NsC+wGfAk4mG5QnQs8FdgXuDjdD3XOA+7YQJ2PUFVLgCUAnU6nNv7ySJL6NZXhdH/P8kPA9htq2Pjl+oWqel2SZwMvBlYn2Q8I3VnXhb07JTkUGBsE1bTfkKuAk4AbgSvozooOBP4rsDtwfVUdOFGdkqSZMZ0XRNwD3J3koOb5a4HLx2uYZI+qurqq3g3cSXeGcyHw+ua0G0n2SvKYZpcDkjypea/pWOC7wNXAIUl2SjIPOL7neMuB05rHa4EXAPdX1T10A2tBkgOb42yV5GlT9zJIkiZruq9AOwE4M8k2wC10Zy/j+ViSPenOfi4F1gBrgYXAqnTPt40CRzXtrwI+DDydbuCcV1UPJ3k7cFnTz7eq6htN+yvoBt7yqnooyT8BPwSoqgeaU4f/M8l2dF+T/wFcP0WvgSRpklI1u94qaU7rnVZVLxl0Lb06nU6NjIwMuoy5JYFZ9vcr6T8kWVlVnfG2DfxzTpIkjTXrPlhaVcuAZQMuQ5I0jZw5afbylJ40tAwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJs1PS/ZE0lAwnSVLrGE6SpNYxnCRJrWM4SZJaZ6jDKclZSfaZoM3SJMeMs35hkldNX3WSpA0Z6nCqqj+rqhs2cfeFgOEkSQMwK8IpyV8meWOz/Ikk32mWD0vyf5IckeSqJKuSnJNkfrN9WZJOs3xKkpuadZ9L8r96DnFwku8luaVnFvVh4KAkq5O8ZQaHK0lz3qwIJ2A5cFCz3AHmJ9kKeD6wDngncHhVPRMYAd7au3OS3wXeBTwH+ENg7zH979L09RK6oQTwNuCKqtqvqj4xXlFJFicZSTIyOjq6mUOUJK03W8JpJbAoybbA/cBVdEPqIODfgX2AK5OsBk4Anjhm/wOAy6vqrqp6EDhnzPbzq+rh5hTg4/stqqqWVFWnqjoLFizYpIFJkn7bloMuoB9V9WCS24CTgO8Ba4EXAHsAtwIXV9XxG+liolsJ3D+JtpKkaTZbZk7QPbV3WvN4BfA6YDWwAnhekqcAJNkmyV5j9v0+cEiSHZJsCRzdx/F+AWw7VcVLkvo3m8LpCrrvDV1VVT8D7qP7ntAocCLw5SRr6YbVI95TqqqfAh8ErgYuAW4A7pngeGuBXydZ4wURkjSzUlWDrmFGJJlfVfc2M6fzgM9X1XlT1X+n06mRkZGp6k4TWX/T1zny9ysNoyQrq6oz3rbZNHPaXO9pLpi4ju77VOcPuB5J0gbMigsipkJVnTboGiRJ/Zkz4aQh4+k8aajNpdN6kqRZwnCSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTW8fuc2mD9V45r8vxeJ2koOXOSJLWO4SRJah3DSZLUOrMinJIsTXJMs3xWkn0muf+901OZJGk6zLoLIqrqz6az/yQBUlUPT+dxJEkbNtCZU5L/lGRtkjVJzktya5Ktmm2PTXLb+uc9+yxL0mmW701yRrP/iiSPb9Y/KclVSa5J8v4x+5/erF+b5L3NuoVJfpDk08AqYLdmtnZdknVJ3jITr4ckqWtg4ZTkacA7gBdW1TOAU4BlwIubJscBX6+qBzfSzWOAFc3+y4FTm/WfBD5TVc8C/l/PMY8A9gQOAPYDFiU5uNn8VOALVbU/sBOwa1XtW1VPB/5mA2NYnGQkycjo6OjkXgBJ0gYNcub0QuBrVXUnQFXdBZwFnNRsP4kNhEKPB4BvNssrgYXN8vOALzfLX+xpf0Tzcy3dGdLedMMK4PaqWtEs3wI8OcmnkhwJ/Nt4B6+qJVXVqarOggULJihVktSvQb7nFOARn6CsqiubU2yHAPOq6roJ+niw6jefwnyIR45nvE9nBvhQVX32ESuThcAve+q4O8kzgD8C/hx4JXDyhCOSJE2JQc6cLgVemeRxAEl2bNZ/ge6sZ6JZ08ZcSfe0IMCre9ZfCJycZH5zzF2T7Dx25yQ7AVtU1deBdwHP3IxaJEmTNLBwqqrrgTOAy5OsAT7ebPpbYAf+47TcpngT8OdJrgG26znmRcCXgKuSrAO+Bmw7zv67AsuSrAaWAm/fjFokSZOUatm9yZrPM72sql476Fomo9Pp1MjIyKbt7L31Nl3L/n4l9S/JyqrqjLetVZ9zSvIp4I+BFw26FknS4LQqnKrqDYOuQZI0eK0KpznLU1OS9Aiz4t56kqS5xXCSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1mnd7YtmqySjwO2DrqPHTsCdgy5iBsyFcc6FMYLjHCb9jvGJVTXu9w0ZTkMqyciG7lk1TObCOOfCGMFxDpOpGKOn9SRJrWM4SZJax3AaXksGXcAMmQvjnAtjBMc5TDZ7jL7nJElqHWdOkqTWMZwkSa1jOA2JJDsmuTjJzc3jDuO02S/JVUmuT7I2ybGDqHWykhyZ5MYkP0rytnG2PyrJV5vtVydZOPNVbr4+xvnWJDc0v7tLkzxxEHVuronG2dPumCSVZNZddt3PGJO8svl9Xp/kSzNd41To42929ySXJbm2+bvt/1vOq8qfIfgBPgq8rVl+G/CRcdrsBezZLP8ucAew/aBrn2Bc84AfA08GfgdYA+wzps1/Ac5slo8DvjrouqdpnC8AtmmWXz+s42zabQssB1YAnUHXPQ2/yz2Ba4Edmuc7D7ruaRrnEuD1zfI+wG399u/MaXi8DDi7WT4bOGpsg6q6qapubpb/Gfg5MO6ns1vkAOBHVXVLVT0AfIXuWHv1jv1rwGFJMoM1ToUJx1lVl1XVr5qnK4Dfm+Eap0I/v0+A99P9D9d9M1ncFOlnjKcCf11VdwNU1c9nuMap0M84C3hss7wd8M/9dm44DY/HV9UdAM3jzhtrnOQAuv/b+fEM1LY5dgX+qef5T5p147apql8D9wCPm5Hqpk4/4+x1CvAP01rR9JhwnEn2B3arqm/OZGFTqJ/f5V7AXkmuTLIiyZEzVt3U6Wec7wFek+QnwLeAN/Tb+ZabW51mTpJLgCeMs+kdk+xnF+CLwAlV9fBU1DaNxpsBjf38Qz9t2q7vMSR5DdABDpnWiqbHRseZZAvgE8CJM1XQNOjnd7kl3VN7h9KdAV+RZN+q+tdprm0q9TPO44GlVfXfkxwIfLEZ54T/7hhOs0hVHb6hbUl+lmSXqrqjCZ9xTxMkeSzw98A7q2rFNJU6lX4C7Nbz/Pf47VMD69v8JMmWdE8f3DUz5U2ZfsZJksPp/mfkkKq6f4Zqm0oTjXNbYF9gWXNm9gnABUleWlUjM1bl5un3b3ZFVT0I3JrkRrphdc3MlDgl+hnnKcCRAFV1VZKt6d4UdsLTmJ7WGx4XACc0yycA3xjbIMnvAOcBX6iqc2awts1xDbBnkic19R9Hd6y9esd+DPCdat6BnUUmHGdzuuuzwEtn6XsUMME4q+qeqtqpqhZW1UK6763NpmCC/v5mz6d7gQtJdqJ7mu+WGa1y8/Uzzn8EDgNI8vvA1sBoX70P+ooPf6bsypnHAZcCNzePOzbrO8BZzfJrgAeB1T0/+w269j7G9iLgJrrvj72jWfc+uv9o0fzBnwP8CPg+8ORB1zxN47wE+FnP7+6CQdc8HeMc03YZs+xqvT5/lwE+DtwArAOOG3TN0zTOfYAr6V7Jtxo4ot++vX2RJKl1PK0nSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrWM4SZJa5/8D7xgtoq1AiKAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import ElasticNet\n",
    "\n",
    "# Create linear regression\n",
    "regressor = ElasticNet(alpha=0.1, l1_ratio=0.1)\n",
    "\n",
    "# Fit/train LASSO\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(f\"Final score (RMSE): {score}\")\n",
    "\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}