lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text classification task\n",
    "\n",
    "As we have mentioned, we will focus on simple text classification task based on **AG_NEWS** dataset, which is to classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech.\n",
    "\n",
    "## The Dataset\n",
    "\n",
    "This dataset is built into [`torchtext`](https://github.com/pytorch/text) module, so we can easily access it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchtext\n",
    "import os\n",
    "import collections\n",
    "os.makedirs('./data',exist_ok=True)\n",
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
    "classes = ['World', 'Sports', 'Business', 'Sci/Tech']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, `train_dataset` and `test_dataset` contain collections that return pairs of label (number of class) and text respectively, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3,\n",
       " \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(train_dataset)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, let's print out the first 10 new headlines from our dataset: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
      "**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
      "**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
      "**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\\flows from the main pipeline in southern Iraq after\\intelligence showed a rebel militia could strike\\infrastructure, an oil official said on Saturday.\n",
      "**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections.\n"
     ]
    }
   ],
   "source": [
    "for i,x in zip(range(5),train_dataset):\n",
    "    print(f\"**{classes[x[0]]}** -> {x[1]}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because datasets are iterators, if we want to use the data multiple times we need to convert it to list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
    "train_dataset = list(train_dataset)\n",
    "test_dataset = list(test_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "\n",
    "Now we need to convert text into **numbers** that can be represented as tensors. If we want word-level representation, we need to do two things:\n",
    "* use **tokenizer** to split text into **tokens**\n",
    "* build a **vocabulary** of those tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['he', 'said', 'hello']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = torchtext.data.utils.get_tokenizer('basic_english')\n",
    "tokenizer('He said: hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = collections.Counter()\n",
    "for (label, line) in train_dataset:\n",
    "    counter.update(tokenizer(line))\n",
    "vocab = torchtext.vocab.vocab(counter, min_freq=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using vocabulary, we can easily encode out tokenized string into a set of numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab size if 95810\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[599, 3279, 97, 1220, 329, 225, 7368]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_size = len(vocab)\n",
    "print(f\"Vocab size if {vocab_size}\")\n",
    "\n",
    "stoi = vocab.get_stoi() # dict to convert tokens to indices\n",
    "\n",
    "def encode(x):\n",
    "    return [stoi[s] for s in tokenizer(x)]\n",
    "\n",
    "encode('I love to play with my words')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of Words text representation\n",
    "\n",
    "Because words represent meaning, sometimes we can figure out the meaning of a text by just looking at the individual words, regardless of their order in the sentence. For example, when classifying news, words like *weather*, *snow* are likely to indicate *weather forecast*, while words like *stocks*, *dollar* would count towards *financial news*.\n",
    "\n",
    "**Bag of Words** (BoW) vector representation is the most commonly used traditional vector representation. Each word is linked to a vector index, vector element contains the number of occurrences of a word in a given document.\n",
    "\n",
    "![Image showing how a bag of words vector representation is represented in memory.](images/bag-of-words-example.png) \n",
    "\n",
    "> **Note**: You can also think of BoW as a sum of all one-hot-encoded vectors for individual words in the text.\n",
    "\n",
    "Below is an example of how to generate a bag of word representation using the Scikit Learn python library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 0, 2, 0, 0, 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    "corpus = [\n",
    "        'I like hot dogs.',\n",
    "        'The dog ran fast.',\n",
    "        'Its hot outside.',\n",
    "    ]\n",
    "vectorizer.fit_transform(corpus)\n",
    "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute bag-of-words vector from the vector representation of our AG_NEWS dataset, we can use the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2., 1., 2.,  ..., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "vocab_size = len(vocab)\n",
    "\n",
    "def to_bow(text,bow_vocab_size=vocab_size):\n",
    "    res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
    "    for i in encode(text):\n",
    "        if i<bow_vocab_size:\n",
    "            res[i] += 1\n",
    "    return res\n",
    "\n",
    "print(to_bow(train_dataset[0][1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** Here we are using global `vocab_size` variable to specify default size of the vocabulary. Since often vocabulary size is pretty big, we can limit the size of the vocabulary to most frequent words. Try lowering `vocab_size` value and running the code below, and see how it affects the accuracy. You should expect some accuracy drop, but not dramatic, in lieu of higher performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training BoW classifier\n",
    "\n",
    "Now that we have learned how to build Bag-of-Words representation of our text, let's train a classifier on top of it. First, we need to convert our dataset for training in such a way, that all positional vector representations are converted to bag-of-words representation. This can be achieved by passing `bowify` function as `collate_fn` parameter to standard torch `DataLoader`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "import numpy as np \n",
    "\n",
    "# this collate function gets list of batch_size tuples, and needs to \n",
    "# return a pair of label-feature tensors for the whole minibatch\n",
    "def bowify(b):\n",
    "    return (\n",
    "            torch.LongTensor([t[0]-1 for t in b]),\n",
    "            torch.stack([to_bow(t[1]) for t in b])\n",
    "    )\n",
    "\n",
    "train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's define a simple classifier neural network that contains one linear layer. The size of the input vector equals to `vocab_size`, and output size corresponds to the number of classes (4). Because we are solving classification task, the final activation function is `LogSoftmax()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will define standard PyTorch training loop. Because our dataset is quite large, for our teaching purpose we will train only for one epoch, and sometimes even for less than an epoch (specifying the `epoch_size` parameter allows us to limit training). We would also report accumulated training accuracy during training; the frequency of reporting is specified using `report_freq` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
    "    net.train()\n",
    "    total_loss,acc,count,i = 0,0,0,0\n",
    "    for labels,features in dataloader:\n",
    "        optimizer.zero_grad()\n",
    "        out = net(features)\n",
    "        loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss+=loss\n",
    "        _,predicted = torch.max(out,1)\n",
    "        acc+=(predicted==labels).sum()\n",
    "        count+=len(labels)\n",
    "        i+=1\n",
    "        if i%report_freq==0:\n",
    "            print(f\"{count}: acc={acc.item()/count}\")\n",
    "        if epoch_size and count>epoch_size:\n",
    "            break\n",
    "    return total_loss.item()/count, acc.item()/count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3200: acc=0.8028125\n",
      "6400: acc=0.8371875\n",
      "9600: acc=0.8534375\n",
      "12800: acc=0.85765625\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.026090790722161722, 0.8620069296375267)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_epoch(net,train_loader,epoch_size=15000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BiGrams, TriGrams and N-Grams\n",
    "\n",
    "One limitation of a bag of words approach is that some words are part of multi word expressions, for example, the word 'hot dog' has a completely different meaning than the words 'hot' and 'dog' in other contexts. If we represent words 'hot` and 'dog' always by the same vectors, it can confuse our model.\n",
    "\n",
    "To address this, **N-gram representations** are often used in methods of document classification, where the frequency of each word, bi-word or tri-word is a useful feature for training classifiers. In bigram representation, for example, we will add all word pairs to the vocabulary, in addition to original words. \n",
    "\n",
    "Below is an example of how to generate a bigram bag of word representation using the Scikit Learn:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary:\n",
      " {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
       "      dtype=int64)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\\b\\w+\\b', min_df=1)\n",
    "corpus = [\n",
    "        'I like hot dogs.',\n",
    "        'The dog ran fast.',\n",
    "        'Its hot outside.',\n",
    "    ]\n",
    "bigram_vectorizer.fit_transform(corpus)\n",
    "print(\"Vocabulary:\\n\",bigram_vectorizer.vocabulary_)\n",
    "bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main drawback of N-gram approach is that vocabulary size starts to grow extremely fast. In practice, we need to combine N-gram representation with some dimensionality reduction techniques, such as *embeddings*, which we will discuss in the next unit.\n",
    "\n",
    "To use N-gram representation in our **AG News** dataset, we need to build special ngram vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bigram vocabulary length =  1308842\n"
     ]
    }
   ],
   "source": [
    "counter = collections.Counter()\n",
    "for (label, line) in train_dataset:\n",
    "    l = tokenizer(line)\n",
    "    counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))\n",
    "    \n",
    "bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
    "\n",
    "print(\"Bigram vocabulary length = \",len(bi_vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could then use the same code as above to train the classifier, however, it would be very memory-inefficient. In the next unit, we will train bigram classifier using embeddings.\n",
    "\n",
    "> **Note:** You can only leave those ngrams that occur in the text more than specified number of times. This will make sure that infrequent bigrams will be omitted, and will decrease the dimensionality significantly. To do this, set `min_freq` parameter to a higher value, and observe the length of vocabulary change."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Term Frequency Inverse Document Frequency TF-IDF\n",
    "\n",
    "In BoW representation, word occurrences are evenly weighted, regardless of the word itself. However, it is clear that frequent words, such as *a*, *in*, etc. are much less important for the classification, than specialized terms. In fact, in most NLP tasks some words are more relevant than others.\n",
    "\n",
    "**TF-IDF** stands for **term frequency–inverse document frequency**. It is a variation of bag of words, where instead of a binary 0/1 value indicating the appearance of a word in a document, a floating-point value is used, which is related to the frequency of word occurrence in the corpus.\n",
    "\n",
    "More formally, the weight $w_{ij}$ of a word $i$ in the document $j$ is defined as:\n",
    "$$\n",
    "w_{ij} = tf_{ij}\\times\\log({N\\over df_i})\n",
    "$$\n",
    "where\n",
    "* $tf_{ij}$ is the number of occurrences of $i$ in $j$, i.e. the BoW value we have seen before\n",
    "* $N$ is the number of documents in the collection\n",
    "* $df_i$ is the number of documents containing the word $i$ in the whole collection\n",
    "\n",
    "TF-IDF value $w_{ij}$ increases proportionally to the number of times a word appears in a document and is offset by the number of documents in the corpus that contains the word, which helps to adjust for the fact that some words appear more frequently than others. For example, if the word appears in *every* document in the collection, $df_i=N$, and $w_{ij}=0$, and those terms would be completely disregarded.\n",
    "\n",
    "You can easily create TF-IDF vectorization of text using Scikit Learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.43381609, 0.        , 0.43381609, 0.        , 0.65985664,\n",
       "        0.43381609, 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        ]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
    "vectorizer.fit_transform(corpus)\n",
    "vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion \n",
    "\n",
    "However even though TF-IDF representations provide frequency weight to different words they are unable to represent meaning or order. As the famous linguist J. R. Firth said in 1935, “The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously.”. We will learn later in the course how to capture contextual information from text using language modeling.\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 ('py38')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}