proc_17_108_analysis_03_level_1_HDBScan_approx_pred.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This notebook is also very short: apply the saved HDBSCAN model to flag remaining clustered items using the function ```approximate_predict()```. This way, we can identify comment groups similar enough to be from campaigns, and comments that are outliers. \n",
    "\n",
    "### Additional Notes\n",
    "\n",
    "* There is a possibility of imprecision with ```approximate_predict()```, such as cluster drift, missing smaller clusters, etc. The assumption we have to make here in order to proceed is that the 100,000 entry random sample is sufficiently representative of the entire population. This may need to be studied further, but the fact that 22 out of 23 million comments were in the top 300 clusters/duplicates makes me comfortable with the assumption at a high level -- needs to be quantified as well.\n",
    "* Warning: I cleaned up the cells but did not re-run them as ```approximate_predict()``` took 12+ hours to finish running. No first-time run guarantee for this notebook only (others I upload I have re-run), but it should be fine.\n",
    "* I will however publish the labeled post-cluster dataset in conjunction with the visualizations notebook, in case you don't feel like waiting overnight for the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import hdbscan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 54.6 s, sys: 2.25 s, total: 56.8 s\n",
      "Wall time: 57.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "X = pd.read_csv('proc_17_108_copy_uniques_level0_unclustered_vectorized.csv', index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "864032"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_sample = X\n",
    "#X_sample = X.sample(1000)\n",
    "len(X_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### hdbscan doesn't support cosine distance so let's use angular distance instead (euclidean of l2 norm vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import normalize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = X_sample.iloc[:,-300:]\n",
    "norm_data = normalize(data, norm='l2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### load our pickled clusterer\n",
    "from sklearn.externals import joblib\n",
    "clusterer = joblib.load('level1-clusterer.pkl') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 12h 8min 14s, sys: 14.2 s, total: 12h 8min 28s\n",
      "Wall time: 12h 11min 19s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "### use our pickled clusterer to predict on new data\n",
    "results, strengths = hdbscan.approximate_predict(clusterer, norm_data)\n",
    "#results = clusterer.#fit_predict(norm_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# immediately save results and strengths to file (by themselves) so we can glom back separately onto the dataframe (don't want to lose 12 hours of computing time!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4 ms, sys: 8 ms, total: 12 ms\n",
      "Wall time: 18.4 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['results-11-19.pkl']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "joblib.dump(results, 'results-11-19.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 0 ns, sys: 8 ms, total: 8 ms\n",
      "Wall time: 10.7 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['strengths-11-19.pkl']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "joblib.dump(strengths, 'strengths-11-19.pkl')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}