Text_Summarization_Pyteaser.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Summarization\n",
    "\n",
    "Text summarization refers to the technique of shortening long pieces of text. The intention is to create a coherent and fluent summary having only the main points outlined in the document.\n",
    "Automatic text summarization is a common problem in machine learning and natural language processing (NLP)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are broadly two different approaches that are used for text summarization:\n",
    "\n",
    "* Extractive Summarization\n",
    "* Abstractive Summarization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will build an extraction based text summarizer using python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extractive Summarization\n",
    "The name gives away what this approach does. We identify the important sentences or phrases from the original text and extract only those from the text. Those extracted sentences would be our summary. The below diagram illustrates extractive summarization:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Extractive Summarization](images/Extractive_Summarization.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTeaser\n",
    "\n",
    "PyTeaser takes any news article and extract a brief summary from it. It's based on the original Scala project [TextTeaer](https://github.com/MojoJolo/textteaser).\n",
    "\n",
    "TextTeaser is an automatic summarization algorithm that combines the power of natural language processing and machine learning to produce good results.\n",
    "\n",
    "Summaries are created by ranking sentences in a news article according to how relevant they are to the entire text. The top 5 sentences are used to form a \"summary\". Each sentence is ranked by using four criteria:\n",
    "\n",
    "* Relevance to the title\n",
    "* Relevance to keywords in the article\n",
    "* Position of the sentence\n",
    "* Length of the sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Summarization using PyTeaser\n",
    "https://github.com/alanbuxton/PyTeaserPython3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the pyteaser.py file from github"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# import package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyteaser import SummarizeUrl,Summarize,keywords\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! wget http://mlg.ucd.ie/files/datasets/bbc-fulltext.zip\n",
    "# ! unzip bbc-fulltext.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"AgustaWestland chopper scam co-accused Rajiv Saxena was extradited to India from UAE on Wednesday. He had been evading the Enforcement Directorate's summons claiming he was suffering from leukaemia but had moved an anti-money laundering court for anticipatory bail in December, stating he had never been summoned at his Dubai address. Saxena's lawyers alleged he had been illegally extradited. \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "title = \"AgustaWestland scam accused Rajiv Saxena extradited to India\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Summarization using PyTeaser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "summaries = Summarize(text,title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['AgustaWestland scam accused Rajiv Saxena extradited to India']\n"
     ]
    }
   ],
   "source": [
    "print(summaries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# url = 'http://www.huffingtonpost.com/2013/11/22/twitter-forward-secrecy_n_4326599.html'\n",
    "# summaries = SummarizeUrl(url)\n",
    "# print(summaries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}