aspect-based-sentiment-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import os\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/kaggle/input'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.chdir('../input')\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "collapsed": true
   },
   "source": [
    "# Aspect Based Sentiment Analysis on Car Reviews\n",
    "## Taking Toyota Cars as an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Review_Date</th>\n",
       "      <th>Author_Name</th>\n",
       "      <th>Vehicle_Title</th>\n",
       "      <th>Review_Title</th>\n",
       "      <th>Review</th>\n",
       "      <th>Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>on 02/02/17 19:53 PM (PST)</td>\n",
       "      <td>Ricardo</td>\n",
       "      <td>1997 Toyota Previa Minivan LE 3dr Minivan</td>\n",
       "      <td>great vehicle, Toyota best design ever. thank you</td>\n",
       "      <td>there is no way back, enjoy what you have .</td>\n",
       "      <td>5.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>on 12/17/16 16:40 PM (PST)</td>\n",
       "      <td>matt</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>my 4th previa, best van ever made!</td>\n",
       "      <td>1st 95 went over 300k before being totalled b...</td>\n",
       "      <td>5.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>on 04/14/10 07:43 AM (PDT)</td>\n",
       "      <td>Joel G</td>\n",
       "      <td>1997 Toyota Previa Minivan LE 3dr Minivan</td>\n",
       "      <td>Mom's Taxi Babies Ride</td>\n",
       "      <td>Sold 86 Toyota Van 285K miles to be replaced ...</td>\n",
       "      <td>5.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>on 11/12/08 17:31 PM (PST)</td>\n",
       "      <td>Dennis</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>My Favorite Van Ever</td>\n",
       "      <td>I have owned lots of vans, and the Previa is ...</td>\n",
       "      <td>4.875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>on 04/14/08 22:47 PM (PDT)</td>\n",
       "      <td>Alf Skrastins</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>Best Minivan ever</td>\n",
       "      <td>My 1997 AWD Previa is the third one that I ha...</td>\n",
       "      <td>5.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Unnamed: 0  ...   Rating\n",
       "0          0  ...    5.000\n",
       "1          1  ...    5.000\n",
       "2          2  ...    5.000\n",
       "3          3  ...    4.875\n",
       "4          4  ...    5.000\n",
       "\n",
       "[5 rows x 7 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_rev = pd.read_csv('../input/Scrapped_Car_Reviews_Toyota.csv',engine='python',index_col=False)\n",
    "toy_rev.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Combining the review title and review body for the text corpus****"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "toy_rev['review']=toy_rev['Review_Title']+toy_rev['Review']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Using spaCy for dependency parsing which forms the crux of aspect extraction**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from tqdm import tqdm\n",
    "nlp = spacy.load('en_core_web_lg', parse=True, tag=True, entity=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## **Using spaCy's awesome displacy module to show the dependency relations**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"91ca3082269543458c7b354a803fefa1-0\" class=\"displacy\" width=\"1100\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Great</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">car</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">and</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">CCONJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">has</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">long</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"925\">range</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"925\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-91ca3082269543458c7b354a803fefa1-0-0\" stroke-width=\"2px\" d=\"M70,177.0 C70,89.5 220.0,89.5 220.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-91ca3082269543458c7b354a803fefa1-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,179.0 L62,167.0 78,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-91ca3082269543458c7b354a803fefa1-0-1\" stroke-width=\"2px\" d=\"M245,177.0 C245,89.5 395.0,89.5 395.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-91ca3082269543458c7b354a803fefa1-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">cc</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M395.0,179.0 L403.0,167.0 387.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-91ca3082269543458c7b354a803fefa1-0-2\" stroke-width=\"2px\" d=\"M245,177.0 C245,2.0 575.0,2.0 575.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-91ca3082269543458c7b354a803fefa1-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">conj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M575.0,179.0 L583.0,167.0 567.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-91ca3082269543458c7b354a803fefa1-0-3\" stroke-width=\"2px\" d=\"M770,177.0 C770,89.5 920.0,89.5 920.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-91ca3082269543458c7b354a803fefa1-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M770,179.0 L762,167.0 778,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-91ca3082269543458c7b354a803fefa1-0-4\" stroke-width=\"2px\" d=\"M595,177.0 C595,2.0 925.0,2.0 925.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-91ca3082269543458c7b354a803fefa1-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M925.0,179.0 L933.0,167.0 917.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "txt = 'Great car and has long range'\n",
    "doc = nlp(txt)\n",
    "spacy.displacy.render(doc,style='dep',jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**from https://nlp.stanford.edu/software/dependencies_manual.pdf**\n",
    "### AMOD - adjectival modifier\n",
    "#### An adjectival modifier of a Noun is any adjectival phrase that serves to modify the meaning of the Noun\n",
    "### ex - 'Great <--amod-- Car', 'Long <--amod-- range'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"24d76d26ad1d451ba27e04c1a66a79ad-0\" class=\"displacy\" width=\"925\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Drives</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">well</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">ADV</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">has</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">great</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">handling</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-24d76d26ad1d451ba27e04c1a66a79ad-0-0\" stroke-width=\"2px\" d=\"M70,177.0 C70,2.0 400.0,2.0 400.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-24d76d26ad1d451ba27e04c1a66a79ad-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,179.0 L62,167.0 78,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-24d76d26ad1d451ba27e04c1a66a79ad-0-1\" stroke-width=\"2px\" d=\"M70,177.0 C70,89.5 220.0,89.5 220.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-24d76d26ad1d451ba27e04c1a66a79ad-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">advmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M220.0,179.0 L228.0,167.0 212.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-24d76d26ad1d451ba27e04c1a66a79ad-0-2\" stroke-width=\"2px\" d=\"M595,177.0 C595,89.5 745.0,89.5 745.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-24d76d26ad1d451ba27e04c1a66a79ad-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M595,179.0 L587,167.0 603,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-24d76d26ad1d451ba27e04c1a66a79ad-0-3\" stroke-width=\"2px\" d=\"M420,177.0 C420,2.0 750.0,2.0 750.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-24d76d26ad1d451ba27e04c1a66a79ad-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M750.0,179.0 L758.0,167.0 742.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "txt = 'Drives well has great handling'\n",
    "doc = nlp(txt)\n",
    "spacy.displacy.render(doc,style='dep',jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ADVMOD - adverb modifier\n",
    "#### An adverb modifier of a word is a (non-clausal) adverb or adverb-headed phrase that serves to modify\n",
    "#### the meaning of the word\n",
    "### ex - 'Drives --advmod--> well'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"8e32bc50adb947d486ef93e932389754-0\" class=\"displacy\" width=\"925\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">wonderful</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">to</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">PART</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">drive</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">the</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">camry</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-8e32bc50adb947d486ef93e932389754-0-0\" stroke-width=\"2px\" d=\"M245,177.0 C245,89.5 395.0,89.5 395.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-8e32bc50adb947d486ef93e932389754-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M245,179.0 L237,167.0 253,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-8e32bc50adb947d486ef93e932389754-0-1\" stroke-width=\"2px\" d=\"M70,177.0 C70,2.0 400.0,2.0 400.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-8e32bc50adb947d486ef93e932389754-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">xcomp</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M400.0,179.0 L408.0,167.0 392.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-8e32bc50adb947d486ef93e932389754-0-2\" stroke-width=\"2px\" d=\"M595,177.0 C595,89.5 745.0,89.5 745.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-8e32bc50adb947d486ef93e932389754-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M595,179.0 L587,167.0 603,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-8e32bc50adb947d486ef93e932389754-0-3\" stroke-width=\"2px\" d=\"M420,177.0 C420,2.0 750.0,2.0 750.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-8e32bc50adb947d486ef93e932389754-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M750.0,179.0 L758.0,167.0 742.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "txt =  \"wonderful to drive the camry \"\n",
    "doc = nlp(txt)\n",
    "spacy.displacy.render(doc,style='dep',jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XCOMP -  open clausal complement\n",
    "#### An open clausal complement (xcomp) of a verb or an adjective is a predicative or clausal complement without its own subject\n",
    "### ex - 'wonderful --xcomp--> drive'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"4aa8f3d22d8f42dab2b2b46c447168f9-0\" class=\"displacy\" width=\"1100\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">not</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">ADV</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">wonderful</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">to</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">PART</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">drive</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">the</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"925\">camry</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"925\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-0\" stroke-width=\"2px\" d=\"M70,177.0 C70,89.5 220.0,89.5 220.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">neg</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,179.0 L62,167.0 78,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-1\" stroke-width=\"2px\" d=\"M420,177.0 C420,89.5 570.0,89.5 570.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M420,179.0 L412,167.0 428,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-2\" stroke-width=\"2px\" d=\"M245,177.0 C245,2.0 575.0,2.0 575.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">xcomp</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M575.0,179.0 L583.0,167.0 567.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-3\" stroke-width=\"2px\" d=\"M770,177.0 C770,89.5 920.0,89.5 920.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M770,179.0 L762,167.0 778,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-4\" stroke-width=\"2px\" d=\"M595,177.0 C595,2.0 925.0,2.0 925.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4aa8f3d22d8f42dab2b2b46c447168f9-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M925.0,179.0 L933.0,167.0 917.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "txt =  \"not wonderful to drive the camry \"\n",
    "doc = nlp(txt)\n",
    "spacy.displacy.render(doc,style='dep',jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NEG - self explanatory\n",
    "### ex - not <--neg-- wonderful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### COMPOUND WORDS\n",
    "#### Generally from a review standpoint, compound words often do not offer us sentiments per se, hence my code looks for possible compound word pairs and then checks with the aspect words extracted if it can add more detail to the extracted aspects - ex Outstanding passenger van gives *more context* than Outstanding van (which is what my code would have extracted without the compound word search) while the compound word search will identify passenger van as a compound word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "competitors = ['Chevy','chevy','Ford','ford','Nissan','nissan','Honda','honda','Chevrolet','chevrolet','Volkswagen','volkswagen','benz','Benz','Mercedes','mercedes','subaru','Subaru','VW']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reason for using competitor name list is to remove potential misleading aspects-sentiments, since we are interested to acquire aspect info about Toyota and not any other brand. This is because a reviewer might be comparing a Benz saying it has superior handling when compared to the car the person is reviewing and this can lead to misclassifications******"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 88%|████████▊ | 20078/22702 [23:57<03:46, 11.59it/s]"
     ]
    }
   ],
   "source": [
    "aspect_terms = []\n",
    "comp_terms = []\n",
    "easpect_terms = []\n",
    "ecomp_terms = []\n",
    "enemy = []\n",
    "for x in tqdm(range(len(toy_rev['review']))):\n",
    "    amod_pairs = []\n",
    "    advmod_pairs = []\n",
    "    compound_pairs = []\n",
    "    xcomp_pairs = []\n",
    "    neg_pairs = []\n",
    "    eamod_pairs = []\n",
    "    eadvmod_pairs = []\n",
    "    ecompound_pairs = []\n",
    "    eneg_pairs = []\n",
    "    excomp_pairs = []\n",
    "    enemlist = []\n",
    "    if len(str(toy_rev['review'][x])) != 0:\n",
    "        lines = str(toy_rev['review'][x]).replace('*',' ').replace('-',' ').replace('so ',' ').replace('be ',' ').replace('are ',' ').replace('just ',' ').replace('get ','').replace('were ',' ').replace('When ','').replace('when ','').replace('again ',' ').replace('where ','').replace('how ',' ').replace('has ',' ').replace('Here ',' ').replace('here ',' ').replace('now ',' ').replace('see ',' ').replace('why ',' ').split('.')       \n",
    "        for line in lines:\n",
    "            enem_list = []\n",
    "            for eny in competitors:\n",
    "                enem = re.search(eny,line)\n",
    "                if enem is not None:\n",
    "                    enem_list.append(enem.group())\n",
    "            if len(enem_list)==0:\n",
    "                doc = nlp(line)\n",
    "                str1=''\n",
    "                str2=''\n",
    "                for token in doc:\n",
    "                    if token.pos_ is 'NOUN':\n",
    "                        for j in token.lefts:\n",
    "                            if j.dep_ == 'compound':\n",
    "                                compound_pairs.append((j.text+' '+token.text,token.text))\n",
    "                            if j.dep_ is 'amod' and j.pos_ is 'ADJ': #primary condition\n",
    "                                str1 = j.text+' '+token.text\n",
    "                                amod_pairs.append(j.text+' '+token.text)\n",
    "                                for k in j.lefts:\n",
    "                                    if k.dep_ is 'advmod': #secondary condition to get adjective of adjectives\n",
    "                                        str2 = k.text+' '+j.text+' '+token.text\n",
    "                                        amod_pairs.append(k.text+' '+j.text+' '+token.text)\n",
    "                                mtch = re.search(re.escape(str1),re.escape(str2))\n",
    "                                if mtch is not None:\n",
    "                                    amod_pairs.remove(str1)\n",
    "                    if token.pos_ is 'VERB':\n",
    "                        for j in token.lefts:\n",
    "                            if j.dep_ is 'advmod' and j.pos_ is 'ADV':\n",
    "                                advmod_pairs.append(j.text+' '+token.text)\n",
    "                            if j.dep_ is 'neg' and j.pos_ is 'ADV':\n",
    "                                neg_pairs.append(j.text+' '+token.text)\n",
    "                        for j in token.rights:\n",
    "                            if j.dep_ is 'advmod'and j.pos_ is 'ADV':\n",
    "                                advmod_pairs.append(token.text+' '+j.text)\n",
    "                    if token.pos_ is 'ADJ':\n",
    "                        for j,h in zip(token.rights,token.lefts):\n",
    "                            if j.dep_ is 'xcomp' and h.dep_ is not 'neg':\n",
    "                                for k in j.lefts:\n",
    "                                    if k.dep_ is 'aux':\n",
    "                                        xcomp_pairs.append(token.text+' '+k.text+' '+j.text)\n",
    "                            elif j.dep_ is 'xcomp' and h.dep_ is 'neg':\n",
    "                                if k.dep_ is 'aux':\n",
    "                                        neg_pairs.append(h.text +' '+token.text+' '+k.text+' '+j.text)\n",
    "            \n",
    "            else:\n",
    "                enemlist.append(enem_list)\n",
    "                doc = nlp(line)\n",
    "                str1=''\n",
    "                str2=''\n",
    "                for token in doc:\n",
    "                    if token.pos_ is 'NOUN':\n",
    "                        for j in token.lefts:\n",
    "                            if j.dep_ == 'compound':\n",
    "                                ecompound_pairs.append((j.text+' '+token.text,token.text))\n",
    "                            if j.dep_ is 'amod' and j.pos_ is 'ADJ': #primary condition\n",
    "                                str1 = j.text+' '+token.text\n",
    "                                eamod_pairs.append(j.text+' '+token.text)\n",
    "                                for k in j.lefts:\n",
    "                                    if k.dep_ is 'advmod': #secondary condition to get adjective of adjectives\n",
    "                                        str2 = k.text+' '+j.text+' '+token.text\n",
    "                                        eamod_pairs.append(k.text+' '+j.text+' '+token.text)\n",
    "                                mtch = re.search(re.escape(str1),re.escape(str2))\n",
    "                                if mtch is not None:\n",
    "                                    eamod_pairs.remove(str1)\n",
    "                    if token.pos_ is 'VERB':\n",
    "                        for j in token.lefts:\n",
    "                            if j.dep_ is 'advmod' and j.pos_ is 'ADV':\n",
    "                                eadvmod_pairs.append(j.text+' '+token.text)\n",
    "                            if j.dep_ is 'neg' and j.pos_ is 'ADV':\n",
    "                                eneg_pairs.append(j.text+' '+token.text)\n",
    "                        for j in token.rights:\n",
    "                            if j.dep_ is 'advmod'and j.pos_ is 'ADV':\n",
    "                                eadvmod_pairs.append(token.text+' '+j.text)\n",
    "                    if token.pos_ is 'ADJ':\n",
    "                        for j in token.rights:\n",
    "                            if j.dep_ is 'xcomp':\n",
    "                                for k in j.lefts:\n",
    "                                    if k.dep_ is 'aux':\n",
    "                                        excomp_pairs.append(token.text+' '+k.text+' '+j.text)\n",
    "        pairs = list(set(amod_pairs+advmod_pairs+neg_pairs+xcomp_pairs))\n",
    "        epairs = list(set(eamod_pairs+eadvmod_pairs+eneg_pairs+excomp_pairs))\n",
    "        for i in range(len(pairs)):\n",
    "            if len(compound_pairs)!=0:\n",
    "                for comp in compound_pairs:\n",
    "                    mtch = re.search(re.escape(comp[1]),re.escape(pairs[i]))\n",
    "                    if mtch is not None:\n",
    "                        pairs[i] = pairs[i].replace(mtch.group(),comp[0])\n",
    "        for i in range(len(epairs)):\n",
    "            if len(ecompound_pairs)!=0:\n",
    "                for comp in ecompound_pairs:\n",
    "                    mtch = re.search(re.escape(comp[1]),re.escape(epairs[i]))\n",
    "                    if mtch is not None:\n",
    "                        epairs[i] = epairs[i].replace(mtch.group(),comp[0])\n",
    "            \n",
    "    aspect_terms.append(pairs)\n",
    "    comp_terms.append(compound_pairs)\n",
    "    easpect_terms.append(epairs)\n",
    "    ecomp_terms.append(ecompound_pairs)\n",
    "    enemy.append(enemlist)\n",
    "toy_rev['compound_nouns'] = comp_terms\n",
    "toy_rev['aspect_keywords'] = aspect_terms\n",
    "toy_rev['competition'] = enemy\n",
    "toy_rev['competition_comp_nouns'] = ecomp_terms\n",
    "toy_rev['competition_aspects'] = easpect_terms\n",
    "toy_rev.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We use vaderSentiment for sentiment analysis because of it's speed and simplicity. It offers 3 types of polarity -  positive, negative and neutral. As a result we can filter all aspects which have high neutral scores hence minimizing errors caused due to wrong extraction of aspects and stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
    "analyser = SentimentIntensityAnalyzer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Review_Date</th>\n",
       "      <th>Author_Name</th>\n",
       "      <th>Vehicle_Title</th>\n",
       "      <th>Review_Title</th>\n",
       "      <th>Review</th>\n",
       "      <th>Rating</th>\n",
       "      <th>review</th>\n",
       "      <th>compound_nouns</th>\n",
       "      <th>aspect_keywords</th>\n",
       "      <th>competition</th>\n",
       "      <th>competition_comp_nouns</th>\n",
       "      <th>competition_aspects</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>on 02/02/17 19:53 PM (PST)</td>\n",
       "      <td>Ricardo</td>\n",
       "      <td>1997 Toyota Previa Minivan LE 3dr Minivan</td>\n",
       "      <td>great vehicle, Toyota best design ever. thank you</td>\n",
       "      <td>there is no way back, enjoy what you have .</td>\n",
       "      <td>5.000</td>\n",
       "      <td>great vehicle, Toyota best design ever. thank ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[great vehicle, is back, best design]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>on 12/17/16 16:40 PM (PST)</td>\n",
       "      <td>matt</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>my 4th previa, best van ever made!</td>\n",
       "      <td>1st 95 went over 300k before being totalled b...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>my 4th previa, best van ever made! 1st 95 went...</td>\n",
       "      <td>[(captain chairs, chairs), (van value, value),...</td>\n",
       "      <td>[minor quirks, red light, fantastic vans, reli...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>on 04/14/10 07:43 AM (PDT)</td>\n",
       "      <td>Joel G</td>\n",
       "      <td>1997 Toyota Previa Minivan LE 3dr Minivan</td>\n",
       "      <td>Mom's Taxi Babies Ride</td>\n",
       "      <td>Sold 86 Toyota Van 285K miles to be replaced ...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>Mom's Taxi Babies Ride Sold 86 Toyota Van 285K...</td>\n",
       "      <td>[(K miles, miles), (reserve weekend, weekend),...</td>\n",
       "      <td>[Use mostly, middle seat, 1st baby, glad to ha...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>on 11/12/08 17:31 PM (PST)</td>\n",
       "      <td>Dennis</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>My Favorite Van Ever</td>\n",
       "      <td>I have owned lots of vans, and the Previa is ...</td>\n",
       "      <td>4.875</td>\n",
       "      <td>My Favorite Van Ever I have owned lots of vans...</td>\n",
       "      <td>[(Fuel mileage, mileage), (Toyota salesman, sa...</td>\n",
       "      <td>[constantly am, ever owned, Ever owned, Never ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>on 04/14/08 22:47 PM (PDT)</td>\n",
       "      <td>Alf Skrastins</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>Best Minivan ever</td>\n",
       "      <td>My 1997 AWD Previa is the third one that I ha...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>Best Minivan ever My 1997 AWD Previa is the th...</td>\n",
       "      <td>[(gas mileage, mileage)]</td>\n",
       "      <td>[previously had, even comes, much fun, reasona...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Unnamed: 0    ...    sentiment\n",
       "0          0    ...          pos\n",
       "1          1    ...          pos\n",
       "2          2    ...          pos\n",
       "3          3    ...          pos\n",
       "4          4    ...          pos\n",
       "\n",
       "[5 rows x 14 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import operator\n",
    "sentiment = []\n",
    "for i in range(len(toy_rev)):\n",
    "    score_dict={'pos':0,'neg':0,'neu':0}\n",
    "    if len(toy_rev['aspect_keywords'][i])!=0: \n",
    "        for aspects in toy_rev['aspect_keywords'][i]:\n",
    "            sent = analyser.polarity_scores(aspects)\n",
    "            score_dict['neg'] += sent['neg']\n",
    "            score_dict['pos'] += sent['pos']\n",
    "        #score_dict['neu'] += sent['neu']\n",
    "        sentiment.append(max(score_dict.items(), key=operator.itemgetter(1))[0])\n",
    "    else:\n",
    "        sentiment.append('NaN')\n",
    "toy_rev['sentiment'] = sentiment\n",
    "toy_rev.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Review_Date</th>\n",
       "      <th>Author_Name</th>\n",
       "      <th>Vehicle_Title</th>\n",
       "      <th>Review_Title</th>\n",
       "      <th>Review</th>\n",
       "      <th>Rating</th>\n",
       "      <th>review</th>\n",
       "      <th>compound_nouns</th>\n",
       "      <th>aspect_keywords</th>\n",
       "      <th>competition</th>\n",
       "      <th>competition_comp_nouns</th>\n",
       "      <th>competition_aspects</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>int_sent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>on 02/02/17 19:53 PM (PST)</td>\n",
       "      <td>Ricardo</td>\n",
       "      <td>1997 Toyota Previa Minivan LE 3dr Minivan</td>\n",
       "      <td>great vehicle, Toyota best design ever. thank you</td>\n",
       "      <td>there is no way back, enjoy what you have .</td>\n",
       "      <td>5.000</td>\n",
       "      <td>great vehicle, Toyota best design ever. thank ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[great vehicle, is back, best design]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>on 12/17/16 16:40 PM (PST)</td>\n",
       "      <td>matt</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>my 4th previa, best van ever made!</td>\n",
       "      <td>1st 95 went over 300k before being totalled b...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>my 4th previa, best van ever made! 1st 95 went...</td>\n",
       "      <td>[(captain chairs, chairs), (van value, value),...</td>\n",
       "      <td>[minor quirks, red light, fantastic vans, reli...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>on 04/14/10 07:43 AM (PDT)</td>\n",
       "      <td>Joel G</td>\n",
       "      <td>1997 Toyota Previa Minivan LE 3dr Minivan</td>\n",
       "      <td>Mom's Taxi Babies Ride</td>\n",
       "      <td>Sold 86 Toyota Van 285K miles to be replaced ...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>Mom's Taxi Babies Ride Sold 86 Toyota Van 285K...</td>\n",
       "      <td>[(K miles, miles), (reserve weekend, weekend),...</td>\n",
       "      <td>[Use mostly, middle seat, 1st baby, glad to ha...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>on 11/12/08 17:31 PM (PST)</td>\n",
       "      <td>Dennis</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>My Favorite Van Ever</td>\n",
       "      <td>I have owned lots of vans, and the Previa is ...</td>\n",
       "      <td>4.875</td>\n",
       "      <td>My Favorite Van Ever I have owned lots of vans...</td>\n",
       "      <td>[(Fuel mileage, mileage), (Toyota salesman, sa...</td>\n",
       "      <td>[constantly am, ever owned, Ever owned, Never ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>on 04/14/08 22:47 PM (PDT)</td>\n",
       "      <td>Alf Skrastins</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>Best Minivan ever</td>\n",
       "      <td>My 1997 AWD Previa is the third one that I ha...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>Best Minivan ever My 1997 AWD Previa is the th...</td>\n",
       "      <td>[(gas mileage, mileage)]</td>\n",
       "      <td>[previously had, even comes, much fun, reasona...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Unnamed: 0                  Review_Date   ...    sentiment int_sent\n",
       "0          0   on 02/02/17 19:53 PM (PST)   ...          pos        1\n",
       "1          1   on 12/17/16 16:40 PM (PST)   ...          pos        1\n",
       "2          2   on 04/14/10 07:43 AM (PDT)   ...          pos        1\n",
       "3          3   on 11/12/08 17:31 PM (PST)   ...          pos        1\n",
       "4          4   on 04/14/08 22:47 PM (PDT)   ...          pos        1\n",
       "\n",
       "[5 rows x 15 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_sent = []\n",
    "for sent in toy_rev['sentiment']:\n",
    "    if sent is 'NaN':\n",
    "        int_sent.append('NaN')\n",
    "    elif sent is 'pos':\n",
    "        int_sent.append('1')\n",
    "    else:\n",
    "        int_sent.append('0')\n",
    "toy_rev['int_sent'] = int_sent\n",
    "toy_rev.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Here we have arbitarily taken ratings greater than 3 as positive and everything else as negative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Review_Date</th>\n",
       "      <th>Author_Name</th>\n",
       "      <th>Vehicle_Title</th>\n",
       "      <th>Review_Title</th>\n",
       "      <th>Review</th>\n",
       "      <th>Rating</th>\n",
       "      <th>review</th>\n",
       "      <th>compound_nouns</th>\n",
       "      <th>aspect_keywords</th>\n",
       "      <th>competition</th>\n",
       "      <th>competition_comp_nouns</th>\n",
       "      <th>competition_aspects</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>int_sent</th>\n",
       "      <th>Positive Review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>on 02/02/17 19:53 PM (PST)</td>\n",
       "      <td>Ricardo</td>\n",
       "      <td>1997 Toyota Previa Minivan LE 3dr Minivan</td>\n",
       "      <td>great vehicle, Toyota best design ever. thank you</td>\n",
       "      <td>there is no way back, enjoy what you have .</td>\n",
       "      <td>5.000</td>\n",
       "      <td>great vehicle, Toyota best design ever. thank ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[great vehicle, is back, best design]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>on 12/17/16 16:40 PM (PST)</td>\n",
       "      <td>matt</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>my 4th previa, best van ever made!</td>\n",
       "      <td>1st 95 went over 300k before being totalled b...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>my 4th previa, best van ever made! 1st 95 went...</td>\n",
       "      <td>[(captain chairs, chairs), (van value, value),...</td>\n",
       "      <td>[minor quirks, red light, fantastic vans, reli...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>on 04/14/10 07:43 AM (PDT)</td>\n",
       "      <td>Joel G</td>\n",
       "      <td>1997 Toyota Previa Minivan LE 3dr Minivan</td>\n",
       "      <td>Mom's Taxi Babies Ride</td>\n",
       "      <td>Sold 86 Toyota Van 285K miles to be replaced ...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>Mom's Taxi Babies Ride Sold 86 Toyota Van 285K...</td>\n",
       "      <td>[(K miles, miles), (reserve weekend, weekend),...</td>\n",
       "      <td>[Use mostly, middle seat, 1st baby, glad to ha...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>on 11/12/08 17:31 PM (PST)</td>\n",
       "      <td>Dennis</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>My Favorite Van Ever</td>\n",
       "      <td>I have owned lots of vans, and the Previa is ...</td>\n",
       "      <td>4.875</td>\n",
       "      <td>My Favorite Van Ever I have owned lots of vans...</td>\n",
       "      <td>[(Fuel mileage, mileage), (Toyota salesman, sa...</td>\n",
       "      <td>[constantly am, ever owned, Ever owned, Never ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>on 04/14/08 22:47 PM (PDT)</td>\n",
       "      <td>Alf Skrastins</td>\n",
       "      <td>1997 Toyota Previa Minivan LE All-Trac 3dr Min...</td>\n",
       "      <td>Best Minivan ever</td>\n",
       "      <td>My 1997 AWD Previa is the third one that I ha...</td>\n",
       "      <td>5.000</td>\n",
       "      <td>Best Minivan ever My 1997 AWD Previa is the th...</td>\n",
       "      <td>[(gas mileage, mileage)]</td>\n",
       "      <td>[previously had, even comes, much fun, reasona...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pos</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Unnamed: 0       ...       Positive Review\n",
       "0          0       ...                     1\n",
       "1          1       ...                     1\n",
       "2          2       ...                     1\n",
       "3          3       ...                     1\n",
       "4          4       ...                     1\n",
       "\n",
       "[5 rows x 16 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import math\n",
    "pos = []\n",
    "for i in range(len(toy_rev)):\n",
    "    if not math.isnan(toy_rev['Rating'][i]):\n",
    "        if int(toy_rev['Rating'][i])>3:\n",
    "            pos.append('1')\n",
    "        else:\n",
    "            pos.append('0')\n",
    "    else:\n",
    "        pos.append('0')\n",
    "toy_rev['Positive Review'] = pos\n",
    "toy_rev.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sent</th>\n",
       "      <th>sent_pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  sent sent_pred\n",
       "0    1         1\n",
       "1    1         1\n",
       "2    1         1\n",
       "3    1         1\n",
       "4    1         1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = {'sent':toy_rev['Positive Review'],'sent_pred':toy_rev['int_sent']}\n",
    "metric_df = pd.DataFrame(data=d)\n",
    "metric_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22702"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(metric_df.sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing NaN values in the sentiment predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17869"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metric_df = metric_df[metric_df.sent_pred != 'NaN']\n",
    "len(metric_df.sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy\n",
      "0.7869494655548716\n",
      "f1 score\n",
      "0.8695026222877319\n",
      "recall\n",
      "0.8829713171818435\n",
      "precision\n",
      "0.8564386521709771\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score,auc,f1_score,recall_score,precision_score\n",
    "print('accuracy')\n",
    "print(accuracy_score(metric_df.sent, metric_df.sent_pred))\n",
    "print('f1 score')\n",
    "print(f1_score(metric_df.sent, metric_df.sent_pred,pos_label='1'))\n",
    "print('recall')\n",
    "print(recall_score(metric_df.sent, metric_df.sent_pred,pos_label='1'))\n",
    "print('precision')\n",
    "print(precision_score(metric_df.sent, metric_df.sent_pred,pos_label='1'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Possible improvements that can be made\n",
    "*  Tricky situation of removing stopwords to reduce unwanted extractions of non-aspects but this can also affect spaCy's dependency parsing. Same goes with noun chunk merging as well. If someone can think of a better way to remove stopwords and still retain spaCy's dependency goodness it can greatly improve the accuracy\n",
    "\n",
    "* This is not a ML task per se since we do more of parsing than ML. Although Bi-Directional LSTM have been very good at ABSA tasks in the past, unlike semeval tasks we do not have a fixed topic for our aspects to fall into. If someone can use the parsing aspect of the code to implement BLSTM in this case, that would be great\n",
    "\n",
    "* better alternatives to vaderSentiment if available (unsupervised/ semi-supervised methods might be better here I think)\n",
    "\n",
    "* The very definition of aspects can be a bit vague at times hence we do not have a valid metric to measure the aspect extraction's accuracy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## P.S This is my first Kaggle Kernel and I am fairly new to python programming as well, hence my non usage of list comprehensions and functions might be evident. I highly encourage everyone to fork my code and add your own twists to increase the accuracy of both aspect extractions and sentiment analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}