01 - Intro to Bookworm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Bookworm\n",
    "## Motivation\n",
    "Infinite Jest is a very long and complicated novel. There are a lot of brilliant resources connected to the book, which aim to help the reader stay afloat amongst the chaos of David Foster Wallace's obscure language, interwoven timelines and narratives, and the sprawling networks of characters. The [Infinite Jest Wiki](http://infinitejest.wallacewiki.com/david-foster-wallace/index.php?title=Infinite_Jest_Page_by_Page), for example, is insanely well documented and I'd recommend it to anyone reading the book.  \n",
    "One of the most interesting resources I found while reading was Sam Potts' [Infinite Jest Diagram](http://www.sampottsinc.com/ij/).  \n",
    "\n",
    "![alt text](https://a.fastcompany.net/upload/IJ_Diagram-Huge-A.jpg \"IJmap\")\n",
    "\n",
    "I went back to the image once or twice while I was reading IJ to work out who a character was and how they were connected to the scene. It's a fun resource to have access to while reading something so deliberately scattered.  \n",
    "However, Infinite Jest isn't the only \"big\" book out there, and as far as I know the network above was drawn up entirely by hand. I thought it would be nice to have something like this for anything I was reading. It might also function as an interesting learning resource - either for kids at a young, early-reader stage with simple books and small character networks, or for people learning about network analysis who have never bothered reading [Les Miserables](https://bl.ocks.org/mbostock/4062045) (again, as far as I know all of the standard example graph datasets like Les Mis and The Karate Kid were put together entirely by hand).  \n",
    "I thought that with a bit of thought and testing, this process was probably automatable, and it is. I can now feed bookworm any novel and have it churn out a pretty network like the one above in seconds, without any prior knowledge of the story or its characters. By virtue of the way character connections are measured, it can also tell you the relative strength of all links between characters.\n",
    "\n",
    "## Getting Started\n",
    "Before we start, let's import all of the code in the [bookworm module](bookworm/). I'll explain what each function does as we move through the notebook - we'll be covering most of [build_network.py](bookworm/build_network.py) here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bookworm import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fisrt thing we'll do is load in a book and a list of its characters. These operations are both pretty simple. The book is loaded in as one long string from a `.txt` file. Character lists are stored in a `.csv`, with all potential names for a character stored on each row. They're loaded in as tuples of names in a list of characters.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "book = load_book('data/raw/ij.txt', lower=True)\n",
    "characters = load_characters('data/raw/characters_ij.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we split the book down into sections. Bookworm works by looking for _coocurrence_ of characters in these sections of the text as a proxy for their connectedness. It's a very simple trick which works stupidly well.  \n",
    "There are a few ways we can break down the book into sections:\n",
    "- `get_sentence_sequences()` uses [NLTK](http://www.nltk.org/)'s standard `.tokenize()` function to split the book into sentences.  \n",
    "- `get_word_sequences()` uses [NLTK](http://www.nltk.org/)'s `word_tokenize()` to split the book into words, of which it will then select ordered lists of length `n` (default 40).  \n",
    "- `get_character_sequences()` uses python builtins to split it into substrings of length `n` (default 200).  \n",
    "\n",
    "Fundamentally, they all return a list of strings which each cover a very small section of the novel. For simplicity's sake we're going to use the sentence-wise splitter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sequences = get_sentence_sequences(book)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now comes the interesting bit. We've assembled our cast, and moved the text that they inhabit into a nice, machine-interpretable format.  \n",
    "What we want to generate now is the blank table below which describes the presence of a character in a sentence. At this point, Bookworm hasn't really 'read' any of the text so all of the interactions between characters and sentences (where each cell in the table represents an interaction) are set to 0:\n",
    "\n",
    "|            | character 1 | character 2 | character 3 |\n",
    "|------------|-------------|-------------|-------------|\n",
    "| sentence 1 | 0           | 0           | 0           |\n",
    "| sentence 2 | 0           | 0           | 0           |\n",
    "| sentence 3 | 0           | 0           | 0           |\n",
    "| sentence 4 | 0           | 0           | 0           |\n",
    "\n",
    "The first bit of the `find_connections()` sets up the blank table above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = find_connections(sequences, characters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, it iterates through the list of sentences it has been fed, checking for an instance of each character. If it finds a character in the sentence, it marks their presence with a 1.  \n",
    "So if **character 1** appears with **character 2** in sentence 1, and with **character 3** in sentence 2, we would see the following, with the rest of the cells remaining blank:\n",
    "\n",
    "|            | character 1 | character 2 | character 3 |\n",
    "|------------|-------------|-------------|-------------|\n",
    "| sentence 1 | 1           | 1           | 0           |\n",
    "| sentence 2 | 1           | 0           | 1           |\n",
    "| sentence 3 | 0           | 0           | 0           |\n",
    "| sentence 4 | 0           | 0           | 0           |\n",
    "\n",
    "In the next stage, we enumerate characters coocurence with one another. We can compute this very quickly by taking the dot product of the table with its transpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cooccurence = calculate_cooccurence(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`calculate_cooccurence()` does this computation and then wipes out any interaction of a character with themselves. For the table above, this would give us:\n",
    "\n",
    "|             | character 1 | character 2 | character 3 |\n",
    "|-------------|-------------|-------------|-------------|\n",
    "| character 1 | 0           | 1           | 1           |\n",
    "| character 2 | 1           | 0           | 0           |\n",
    "| character 3 | 1           | 0           | 0           |\n",
    "\n",
    "showing that **character 1** has interacted with **character 2** and **character 3**, but **character 2** and **character 3** haven't interacted. Note the symmetry across the diagonal...\n",
    "\n",
    "The cooccurence matrix we're referring to here is also known as an _adjacency matrix_ - I might use the terms interchangably from here on. \n",
    "\n",
    "The example table above is miniscule in comparison to the dozens of characters who might turn up in a reasonably sized novel, and the hundreds or thousands of opportunities they have to interact with one another. The coocurence matrix in reality is likely to contain much larger numbers between characters who regularly appear in the same sentences. Unless we're working with a _really_ tiny, incestuous network, this coocurence matrix is also probably going to be pretty sparse. For that reason it'll often make sense to store it as a sparse matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cooccurence = cooccurence.to_sparse()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's the essence of what bookworm does, and everything from here onwards is just play. It really is that simple. Once we have an adjacency matrix of our characters, all of the graph theory falls into place.\n",
    "\n",
    "So, now we can show off a few some results! Despite describing a set of tiny matrices above, we've really been computing all of Infinite Jest's massiveness while working through the notebook.\n",
    "\n",
    "We can print the strongest relationships for a chosen character using the function below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def print_five_closest(character):\n",
    "    print('-'*len(str(character))\n",
    "          + '\\n' + str(character) + '\\n'\n",
    "          + '-'*len(str(character)))\n",
    "    \n",
    "    top_five = (cooccurence[str(character)]\n",
    "                .sort_values(ascending=False)\n",
    "                .index.values\n",
    "                [:5])\n",
    "    \n",
    "    for name in top_five:\n",
    "        print(name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying this to 5 characters at random:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------\n",
      "('joubet ',)\n",
      "------------\n",
      "('marathe ', 'remy ')\n",
      "('desjardins ',)\n",
      "('zoltan csikzentmihalyi ',)\n",
      "('fdv ', 'harde ', 'fall down very ')\n",
      "('gavin diehl ', 'gavin ', 'diehl ')\n",
      "\n",
      "----------------------------------------------------\n",
      "('guillaume duplessis ', 'guillaume ', 'duplessis ')\n",
      "----------------------------------------------------\n",
      "('marathe ', 'remy ')\n",
      "('steeply ', 'hugh ')\n",
      "('fortier ',)\n",
      "('luria perec ', 'luria p ')\n",
      "('zoltan csikzentmihalyi ',)\n",
      "\n",
      "-------------------------------------\n",
      "('the moms ', 'avril ', 'mondragon ')\n",
      "-------------------------------------\n",
      "('hal ',)\n",
      "('orin ',)\n",
      "('mario ',)\n",
      "('himself ', 'mad stork ', 'jim icandenza ', 'james incandenza ')\n",
      "('joelle ', 'van dyne ', 'lucille ')\n",
      "\n",
      "-------------\n",
      "('dymphna ',)\n",
      "-------------\n",
      "('petropolis khan ', 'petropolis ')\n",
      "('zoltan csikzentmihalyi ',)\n",
      "('evan ingersoll ', 'ingersoll ')\n",
      "('gately ', 'don ')\n",
      "('fully functional phil ',)\n",
      "\n",
      "------------------------\n",
      "('dean of admissions ',)\n",
      "------------------------\n",
      "('zoltan csikzentmihalyi ',)\n",
      "('dolores epps ',)\n",
      "('gavin diehl ', 'gavin ', 'diehl ')\n",
      "('gately ', 'don ')\n",
      "('fully functional phil ',)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from random import randint\n",
    "\n",
    "for i in range(5):\n",
    "    print_five_closest(characters[randint(0, len(characters))])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those all seem to make sense... Lets try with a few characters who we know about in more detail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------\n",
      "('the moms ', 'avril ', 'mondragon ')\n",
      "-------------------------------------\n",
      "('hal ',)\n",
      "('orin ',)\n",
      "('mario ',)\n",
      "('himself ', 'mad stork ', 'jim icandenza ', 'james incandenza ')\n",
      "('joelle ', 'van dyne ', 'lucille ')\n"
     ]
    }
   ],
   "source": [
    "print_five_closest(('the moms ', 'avril ', 'mondragon '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------\n",
      "('joelle ', 'van dyne ', 'lucille ')\n",
      "------------------------------------\n",
      "('orin ',)\n",
      "('gately ', 'don ')\n",
      "('the moms ', 'avril ', 'mondragon ')\n",
      "('erdedy ',)\n",
      "('himself ', 'mad stork ', 'jim icandenza ', 'james incandenza ')\n"
     ]
    }
   ],
   "source": [
    "print_five_closest(('joelle ', 'van dyne ', 'lucille '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------\n",
      "('pemulis ',)\n",
      "-------------\n",
      "('hal ',)\n",
      "('trevor \"axhandle\" axford ', 'axford ', 'axhandle ')\n",
      "('jim troeltsch ', 'troeltsch ')\n",
      "('james struck ', 'struck ')\n",
      "('keith freer ', 'freer ', 'the viking ')\n"
     ]
    }
   ],
   "source": [
    "print_five_closest(('pemulis ',))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------\n",
      "('bruce green ',)\n",
      "-----------------\n",
      "('randy ', 'lenz ')\n",
      "('gately ', 'don ')\n",
      "('himself ', 'mad stork ', 'jim icandenza ', 'james incandenza ')\n",
      "('kate gompert ', 'gompert ')\n",
      "('tommy doocey ',)\n"
     ]
    }
   ],
   "source": [
    "print_five_closest(('bruce green ',))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yep... Compare the results we've generated to the ones in the diagram at the top of the notebook.\n",
    "\n",
    "# Same code, different book\n",
    "Lets run the whole thing for an entirely different book and see whether we get similarly positive results. This time, Harry Potter and The Philosopher's Stone - chosen because you're more likely to have some contextual knowledge of who's who and what's what in that book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "book = load_book('data/raw/hp_philosophers_stone.txt', lower=True)\n",
    "characters = load_characters('data/raw/characters_hp.csv')\n",
    "sequences = get_sentence_sequences(book)\n",
    "\n",
    "df = find_connections(sequences, characters)\n",
    "cooccurence = calculate_cooccurence(df).to_sparse()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('vernon ', ' dursley '),\n",
       " ('petunia ', ' dursley '),\n",
       " ('dudley ', ' duddy '),\n",
       " ('lily ',),\n",
       " ('james ',)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "characters[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------\n",
      "('harry ', ' potter ')\n",
      "----------------------\n",
      "('ron ', ' weasley ')\n",
      "('hermione ', ' granger ')\n",
      "('hagrid ', ' rubeus ')\n",
      "('snape ', ' severus ')\n",
      "('dudley ', ' duddy ')\n"
     ]
    }
   ],
   "source": [
    "print_five_closest(('harry ', ' potter '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------\n",
      "('voldemort ', ' lord ', ' you-know-who ')\n",
      "------------------------------------------\n",
      "('harry ', ' potter ')\n",
      "('snape ', ' severus ')\n",
      "('quirrell ',)\n",
      "('dumbledore ', ' albus ')\n",
      "('ron ', ' weasley ')\n"
     ]
    }
   ],
   "source": [
    "print_five_closest(('voldemort ', ' lord ', ' you-know-who '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------\n",
      "('crabbe ',)\n",
      "------------\n",
      "('goyle ',)\n",
      "('draco ', ' malfoy ')\n",
      "('harry ', ' potter ')\n",
      "('neville ', ' longbottom ')\n",
      "('george ',)\n"
     ]
    }
   ],
   "source": [
    "print_five_closest(('crabbe ',))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------\n",
      "('fred ',)\n",
      "----------\n",
      "('george ',)\n",
      "('ron ', ' weasley ')\n",
      "('harry ', ' potter ')\n",
      "('adrian pucey ',)\n",
      "('katie bell ',)\n"
     ]
    }
   ],
   "source": [
    "print_five_closest(('fred ',))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully that's enough proof that bookworm does its job well.  \n",
    "In the next notebook we'll examine how we can automatically extract character names from novels in order to automate the entirity of the bookworm process.\n",
    "\n",
    "[Home](https://github.com/harrisonpim/bookworm) | [02 - Character Building >](02%20-%20Character%20Building.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}