9. Understanding VideoBERT, BART, and more/9.11. Computing contextual word representation .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cRJwxgQk2Vr1"
   },
   "source": [
    "# Computing contextual word representation \n",
    "In the previous section, we learned how to use bert-as-service for obtaining the fixed-length sentence representation. In this section, let us learn how to use the bert-as-service to obtained a contextual word representation.\n",
    "\n",
    "\n",
    "#### Make sure to run this notebook in GPU \n",
    "\n",
    "Install the libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "id": "I8yNdff32q4K"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install tensorflow==1.14\n",
    "!pip install bert-serving-client\n",
    "!pip install -U bert-serving-server[http]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xmoN9T2H3G6J"
   },
   "source": [
    "\n",
    "We know that BERT model returns the representation of each word in the sentence and the representation of the word is based on the context of the word used in the sentence. To obtain the word representation, we set the pooling strategy to NONE while starting the BERT server. We also pass the max sequence length as an argument. Each sentence varies in length right? Yes. Thus we set the maximum sequence length to 20 as given below: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-IumfeiB2VsM",
    "outputId": "18c9c8b8-b577-41ee-bbcc-52caa3250cf0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-12-30 23:08:14--  https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\n",
      "Resolving storage.googleapis.com (storage.googleapis.com)... 172.253.122.128, 142.250.73.208, 172.253.63.128, ...\n",
      "Connecting to storage.googleapis.com (storage.googleapis.com)|172.253.122.128|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 407727028 (389M) [application/zip]\n",
      "Saving to: ‘uncased_L-12_H-768_A-12.zip’\n",
      "\n",
      "uncased_L-12_H-768_ 100%[===================>] 388.84M   311MB/s    in 1.3s    \n",
      "\n",
      "2020-12-30 23:08:16 (311 MB/s) - ‘uncased_L-12_H-768_A-12.zip’ saved [407727028/407727028]\n",
      "\n",
      "Archive:  uncased_L-12_H-768_A-12.zip\n",
      "   creating: uncased_L-12_H-768_A-12/\n",
      "  inflating: uncased_L-12_H-768_A-12/bert_model.ckpt.meta  \n",
      "  inflating: uncased_L-12_H-768_A-12/bert_model.ckpt.data-00000-of-00001  \n",
      "  inflating: uncased_L-12_H-768_A-12/vocab.txt  \n",
      "  inflating: uncased_L-12_H-768_A-12/bert_model.ckpt.index  \n",
      "  inflating: uncased_L-12_H-768_A-12/bert_config.json  \n"
     ]
    }
   ],
   "source": [
    "!wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\n",
    "!unzip uncased_L-12_H-768_A-12.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "id": "fdXxnSfJ3j_z"
   },
   "outputs": [],
   "source": [
    "!nohup bert-serving-start -max_seq_len=20 -pooling_strategy NONE -model_dir=./uncased_L-12_H-768_A-12 > out.file 2>&1 &"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WsHTSlt_2VsR"
   },
   "source": [
    "Import the BERT client:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "id": "i5rFONBA2VsT"
   },
   "outputs": [],
   "source": [
    "from bert_serving.client import BertClient\n",
    "bc = BertClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BiJ-9IMK2VsU"
   },
   "source": [
    "\n",
    "Define the sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "id": "V_9WJUnW2VsW"
   },
   "outputs": [],
   "source": [
    "sentence = 'The weather is great today'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YaGw1jCm2VsX"
   },
   "source": [
    "\n",
    "Next, compute the vector representation of the sentence: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "id": "QiQkf-PZ2VsZ"
   },
   "outputs": [],
   "source": [
    "vec = bc.encode([sentence])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_KyV1VKx2Vsb"
   },
   "source": [
    "\n",
    "Let us check the size of the vector: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jHuqUixT2Vsc",
    "outputId": "f1dee0aa-7ec7-4d3b-f52b-ead251829c3d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 20, 768)\n"
     ]
    }
   ],
   "source": [
    "print(vec.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4rvYk8mb2Vsd"
   },
   "source": [
    "\n",
    "As you can notice, unlike what we saw in the previous section, here the size of the given sentence is (1,20,768). This basically implies that we have a representation for each word in the given sentence. That is, we know that in the BERT model, we use the [CLS] token at the beginning and [SEP] token at the end of the sentence, \n",
    "\n",
    "- vec[0][0] - holds the representation of the token [CLS] \n",
    "- vec[0][1] - holds the representation of the first word in the sentence 'the' \n",
    "- vec[0][2] - holds the representation of the second word in the sentence 'weather' \n",
    "- vec[0][3] - holds the representation of the third word in the sentence  'is' \n",
    "- vec[0][4] - holds the representation of the fourth word in the sentence 'great' \n",
    "- vec[0][5] - holds the representation of the fifth word in the sentence 'today' \n",
    "- vec[0][6] - holds the representation of the [SEP] token. \n",
    "- vec[0][7] to vec[0][20] - holds the representation of the padding tokens \n",
    "\n",
    "In this way, we can use bert-as-service for various use cases. "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "9.11. Computing contextual word representation .ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}