src/02_Training_Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Models 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVM-Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "“Support Vector Machine” (SVM) is a supervised machine learning algorithm which can be used for either classification or regression problems. However, it is mostly used in classification. In this algorithm, we plot each data item as a point in n-dimensional space (where n is the number of features you have) called support vectors. Then, we perform classification by finding the hyperplane that differentiates the two classes. Additionally, SVM is robust to outliers and has a feature to ignore them.\n",
    "\n",
    "SVM constructs a hyperplane in multidimensional space to separate different classes. SVM generates optimal hyperplane in an iterative manner, which is used to minimize an error. The core idea of SVM is to find a maximum marginal hyperplane(MMH) that best divides the dataset into classes.\n",
    "\n",
    "- ref (https://engmrk.com/module-16-support-vector-machine-2/?utm_campaign=News&utm_medium=Community&utm_source=DataCamp.com_)\n",
    "- (https://datacamp.com/community/tutorials/svm-classification-scikit-learn-python)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support Vectors\n",
    "- Support vectors are the data points, which are closest to the hyperplane. These points will define the separating line better by calculating margins. These points are more relevant to the construction of the classifier.\n",
    "\n",
    "### Hyperplane\n",
    "- A hyperplane is a decision plane which separates between a set of objects having different class memberships.\n",
    "- Plane which is of n-1 dimension where n is dimension of data.\n",
    "\n",
    "###  Margin\n",
    "- A margin is a gap between the two lines on the closest class points. This is calculated as the perpendicular distance from the line to support vectors or closest points. If the margin is larger in between the classes, then it is considered a good margin, a smaller margin is a bad margin.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How does SVM work?\n",
    "The main objective is to segregate the given dataset in the best possible way. The distance between the either nearest points is known as the margin. The objective is to select a hyperplane with the maximum possible margin between support vectors in the given dataset. SVM searches for the maximum marginal hyperplane in the following steps:\n",
    "\n",
    "1. Generate hyperplanes which segregates the classes in the best way. Left-hand side figure showing three hyperplanes black, blue and orange. Here, the blue and orange have higher classification error, but the black is separating the two classes correctly.\n",
    "\n",
    "2. Select the right hyperplane with the maximum segregation from the either nearest data points as shown in the right-hand side figure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM1](img/svm.webp)\n",
    "\n",
    "\n",
    "Dealing with non-linear and inseparable planes\n",
    "Some problems can’t be solved using linear hyperplane, as shown in the figure below (left-hand side).\n",
    "\n",
    "In such situation, SVM uses a kernel trick to transform the input space to a higher dimensional space as shown on the right. The data points are plotted on the x-axis and z-axis (Z is the squared sum of both x and y: z=x^2=y^2). Now you can easily segregate these points using linear separation.\n",
    "\n",
    "![SVM2](img/svm2.webp)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of SVM kernels\n",
    "\n",
    "- Linear Kernel:A linear kernel can be used as normal dot product any two given observations. The product between two vectors is the sum of the multiplication of each pair of input values.\n",
    "\n",
    "- Polynomial Kernel:A polynomial kernel is a more generalized form of the linear kernel. The polynomial kernel can distinguish curved or nonlinear input space\n",
    "\n",
    "- Radial Basis Function Kernel: The Radial basis function kernel is a popular kernel function commonly used in support vector machine classification. RBF can map an input space in infinite dimensional space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM Parameters\n",
    "\n",
    "* C\n",
    "* C is used during the training phase and says how much outliers are taken into account in calculating Support Vectors.\n",
    "\n",
    "* A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly by giving the model freedom to select more samples as support vectors.\n",
    "\n",
    "* Gamma\n",
    "* The gamma parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’.\n",
    "\n",
    "* The gamma parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.\n",
    "\n",
    "* Kernel\n",
    "* This parameter has many options like, “linear”, “rbf”,”poly” and others (default value is “rbf”). ‘linear’ is used for linear hyper-plane whereas “rbf” and “poly” are used for non-linear hyper-plane.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Soft Margin Classification\n",
    "\n",
    "If we strictly impose that all instances be off the street and on the right side, this is\n",
    "called hard margin classification. There are two main issues with hard margin classification. \n",
    "\n",
    "-  it only works if the data is linearly separable.\n",
    "-  it is quite sensitive to outliers.\n",
    "\n",
    "To avoid these issues it is preferable to use a more flexible model. The objective is to find a good balance between keeping the street as large as possible and limiting the margin violations (i.e., instances that end up in the middle of the street or even on the wrong side). This is called ___soft margin classification.___\n",
    "\n",
    "![SVM-SoftMargin](img/svm3.PNG)\n",
    "\n",
    "In Scikit-Learn’s SVM classes, you can control this balance using the C hyperparameter: a smaller C value leads to a wider street but more margin violations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('scaler',\n",
       "                 StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
       "                ('linear_svc',\n",
       "                 LinearSVC(C=1, class_weight=None, dual=True,\n",
       "                           fit_intercept=True, intercept_scaling=1,\n",
       "                           loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "                           penalty='l2', random_state=None, tol=0.0001,\n",
       "                           verbose=0))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import LinearSVC\n",
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)] # petal length, petal width\n",
    "y = (iris[\"target\"] == 2).astype(np.float64) # Iris-Virginica\n",
    "svm_clf = Pipeline((\n",
    " (\"scaler\", StandardScaler()),\n",
    " (\"linear_svc\", LinearSVC(C=1, loss=\"hinge\")),\n",
    " ))\n",
    "svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_clf.predict([[5.5, 1.7]])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- using SVC(kernel=\"linear\", C=1) is also possible, but it is much slower, especially with large training sets, so it is not recommended.\n",
    "- The SGDClassifier class, with SGDClassifier(loss=\"hinge\",alpha=1/(m*C)). This applies regular Stochastic Gradient Descent to train a linear SVM classifier. It does not converge as fast as the LinearSVC class, but it can be useful to handle huge datasets that do not fit in memory (out-of-core training), or to handle online classification tasks.\n",
    "\n",
    "> The LinearSVC class regularizes the bias term, so you should center\n",
    "the training set first by subtracting its mean. This is automatic if\n",
    "you scale the data using the StandardScaler. Moreover, make sure\n",
    "you set the loss hyperparameter to \"hinge\", as it is not the default\n",
    "value. Finally, for better performance you should set the dual\n",
    "hyperparameter to False, unless there are more features than\n",
    "training instances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nonlinear SVM Classification\n",
    "\n",
    "Although linear SVM classifiers are efficient and work surprisingly well in many\n",
    "cases, many datasets are not even close to being linearly separable. One approach to handling nonlinear datasets is to add more features, such as polynomial features.\n",
    "\n",
    "Consider the left plot in Figure it represents a simple dataset with just one feature x1. This dataset is not linearly separable, as you can see. But if you add a second feature $x_2 = (x_1)^2$, the resulting 2D dataset is perfectly linearly separable.\n",
    "\n",
    "![SVM-NonLinear](img/svm4.PNG)\n",
    "\n",
    "\n",
    "```python\n",
    "from sklearn.datasets import make_moons\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "polynomial_svm_clf = Pipeline((\n",
    " (\"poly_features\", PolynomialFeatures(degree=3)),\n",
    " (\"scaler\", StandardScaler()),\n",
    " (\"svm_clf\", LinearSVC(C=10, loss=\"hinge\"))\n",
    " ))\n",
    "polynomial_svm_clf.fit(X, y)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polynomial Kernel \n",
    "\n",
    "Adding polynomial features is simple to implement and can work great with all sorts of Machine Learning algorithms (not just SVMs), but at a low polynomial degree it cannot deal with very complex datasets, and with a high polynomial degree it creates a huge number of features, making the model too slow.\n",
    "Fortunately, when using SVMs you can apply an almost miraculous mathematical\n",
    "technique called the kernel trick (it is explained in a moment). It makes it possible to get the same result as if you added many polynomial features, even with very highdegree polynomials, without actually having to add them. So there is no combinatorial explosion of the number of features since you don’t actually add any features. This trick is implemented by the SVC class. \n",
    "\n",
    "```python\n",
    "from sklearn.svm import SVC\n",
    "poly_kernel_svm_clf = Pipeline((\n",
    " (\"scaler\", StandardScaler()),\n",
    " (\"svm_clf\", SVC(kernel=\"poly\", degree=3, coef0=1, C=5))\n",
    " ))\n",
    "poly_kernel_svm_clf.fit(X, y)\n",
    "```\n",
    "\n",
    "> A common approach to find the right hyperparameter values is to\n",
    "use grid search. It is often faster to first do a very\n",
    "coarse grid search, then a finer grid search around the best values\n",
    "found. Having a good sense of what each hyperparameter actually\n",
    "does can also help you search in the right part of the hyperparameter space.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Similarity Features\n",
    "\n",
    "Another technique to tackle nonlinear problems is to add features computed using a\n",
    "similarity function that measures how much each instance resembles a particular\n",
    "landmark.\n",
    "\n",
    "let’s define the similarity function to be the Gaussian Radial Basis Function (RBF)\n",
    "with γ = 0.3\n",
    "\n",
    "$$ Gaussian RBF $$\n",
    "\n",
    "$$\\phi y(x,l)=exp(-y|x-l|^2)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gaussian RBF Kernel\n",
    "\n",
    "Just like the polynomial features method, the similarity features method can be useful\n",
    "with any Machine Learning algorithm, but it may be computationally expensive to\n",
    "compute all the additional features, especially on large training sets. However, once\n",
    "again the kernel trick does its SVM magic: it makes it possible to obtain a similar\n",
    "result as if you had added many similarity features, without actually having to add\n",
    "them. \n",
    "```python\n",
    "rbf_kernel_svm_clf = Pipeline((\n",
    " (\"scaler\", StandardScaler()),\n",
    " (\"svm_clf\", SVC(kernel=\"rbf\", gamma=5, C=0.001))\n",
    " ))\n",
    "rbf_kernel_svm_clf.fit(X, y)\n",
    "```\n",
    "\n",
    "This model is represented on the bottom left of Figure. The other plots show\n",
    "models trained with different values of hyperparameters gamma (γ) and C. Increasing\n",
    "gamma makes the bell-shape curve narrower , and as a\n",
    "result each instance’s range of influence is smaller: the decision boundary ends up being more irregular, wiggling around individual instances. Conversely, a small gamma value makes the bell-shaped curve wider, so instances have a larger range of influence, and the decision boundary ends up smoother. So γ acts like a regularization hyperparameter: if your model is overfitting, you should reduce it, and if it is underfitting, you should increase it (similar to the C hyperparameter).\n",
    "\n",
    "![](img/svm5.PNG)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___IMPORTANT___\n",
    "\n",
    "> With so many kernels to choose from, how can you decide which\n",
    "one to use? As a rule of thumb, you should always try the linear\n",
    "kernel first (remember that LinearSVC is much faster than SVC(ker\n",
    "nel=\"linear\")), especially if the training set is very large or if it\n",
    "has plenty of features. If the training set is not too large, you should\n",
    "try the Gaussian RBF kernel as well; it works well in most cases.\n",
    "Then if you have spare time and computing power, you can also\n",
    "experiment with a few other kernels using cross-validation and grid\n",
    "search, especially if there are kernels specialized for your training\n",
    "set’s data structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computational Complexity\n",
    "\n",
    "<table style =\"width:100%\">\n",
    "    <tr>\n",
    "        <th>Class</th> \n",
    "        <th>Time complexity</th>           <th>Out-of-core support</th> \n",
    "        <th>Scaling required</th> <th>Kernel trick</th>\n",
    "    </tr>\n",
    "    <tr><td>LinearSVC</td><td> O(m × n)</td><td> No</td><td> Yes</td><td> No</td></tr>\n",
    "    \n",
    "<tr><td>SGDClassifier</td><td> O(m × n)</td><td> Yes</td><td> Yes</td><td> No</td></tr>\n",
    "<tr><td>SVC</td><td> O(m² × n) to O(m³ × n)</td><td> No</td><td> Yes</td><td> Yes</td></tr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM Regression\n",
    "\n",
    "```python\n",
    "from sklearn.svm import LinearSVR\n",
    "svm_reg = LinearSVR(epsilon=1.5)\n",
    "svm_reg.fit(X, y)\n",
    "\n",
    "from sklearn.svm import LinearSVR\n",
    "svm_reg = LinearSVR(epsilon=1.5)\n",
    "svm_reg.fit(X, y)\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Add Under The Hood Theory for SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}