a-detailed-explanation-of-keras-embedding-layer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d3d6451a0f6029825c07e3efadcd77169b31db9e"
   },
   "source": [
    "## A Detailed Guide to understand the Word Embeddings and Embedding Layer in Keras."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7acfc8938e6bc84b3fc3f458cb74e7aeb296e12d"
   },
   "source": [
    "## [Don't forget to upvote ;) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "3017fc7c972715b9236e2e0d77b74e1b36dbe8e9"
   },
   "source": [
    "In this kernel I have explained the keras embedding layer. To do so I have created a sample corpus of just 3 documents and that should be sufficient to explain the working of the keras embedding layer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ed3010328fb2965b8c7665d54bb5690bbf3a9714"
   },
   "source": [
    "Embeddings are useful in a variety of machine learning applications. Because of the fact I have attached many data sources to the kernel where I fell that embeddings and Keras embedding layer may prove to be useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7a5c0786e978d93aa5de2a4315cb817d66f3da38"
   },
   "source": [
    "Before diving in let us skim through some of the applilcations of the embeddings : \n",
    "\n",
    "**1 ) The first application that strikes me is in the Collaborative Filtering based Recommender Systems where we have to create the user embeddings and the movie embeddings by decomposing the utility matrix which contains the user-item ratings.**\n",
    "\n",
    "To see a complete tutorial on CF based recommender systems using embeddings in Keras you can follow **[this](https://www.kaggle.com/rajmehra03/cf-based-recsys-by-low-rank-matrix-factorization)** kernel of mine.\n",
    "\n",
    "\n",
    "**2 ) The second use is in the Natural Language Processing and its related applications whre we have to create the word embeddings for all the words present in the documents of our corpus.**\n",
    "\n",
    "This is the terminology that I shall use in this kernel.\n",
    "\n",
    "\n",
    "**Thus the embedding layer in Keras can be used when we want to create the embeddings to embed higher dimensional data into lower dimensional vector space.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d48312c5146ad38bab97d505dd35a147b946de4a"
   },
   "source": [
    "#### IMPORTING MODULES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\g.gagliano\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers\\punkt.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow.keras as keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_uuid": "cedb06de49a94e693e8de60941b7c57d6997f63f"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# Ignore  the warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('always')\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# data visualisation and manipulation\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import style\n",
    "import seaborn as sns\n",
    "#configure\n",
    "# sets matplotlib to inline and displays graphs below the corressponding cell.\n",
    "%matplotlib inline  \n",
    "style.use('fivethirtyeight')\n",
    "sns.set(style='whitegrid',color_codes=True)\n",
    "\n",
    "#nltk\n",
    "import nltk\n",
    "\n",
    "#stop-words\n",
    "from nltk.corpus import stopwords\n",
    "stop_words=set(nltk.corpus.stopwords.words('english'))\n",
    "\n",
    "# tokenizing\n",
    "from nltk import word_tokenize,sent_tokenize\n",
    "\n",
    "#keras\n",
    "from keras.preprocessing.text import one_hot,Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense , Flatten ,Embedding,Input\n",
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "604b9fdec82053d0170e98af3e76594001ee9e43"
   },
   "source": [
    "#### CREATING SAMPLE CORPUS OF DOCUMENTS ie TEXTS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "b789f1783a653c9a6b74541fd27348f5680c6569"
   },
   "outputs": [],
   "source": [
    "sample_text_1=\"bitty bought a bit of butter\"\n",
    "sample_text_2=\"but the bit of butter was a bit bitter\"\n",
    "sample_text_3=\"so she bought some better butter to make the bitter butter better\"\n",
    "\n",
    "corp=[sample_text_1,sample_text_2,sample_text_3]\n",
    "no_docs=len(corp)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2fc3f964bec6dab85e0b9e5874d065f6f5637f1d"
   },
   "source": [
    "#### INTEGER ENCODING ALL THE DOCUMENTS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "46bd3ac6a6dfb9e73708302c20cd7091092b596e"
   },
   "source": [
    "After this all the unique words will be reprsented by an integer. For this we are using **one_hot** function from the Keras. Note that the **vocab_size**  is specified large enough so as to ensure **unique integer encoding**  for each and every word.\n",
    "\n",
    "**Note one important thing that the integer encoding for the word remains same in different docs. eg 'butter' is  denoted by 31 in each and every document.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_uuid": "04453f82e7bd7cb51c8a2c7d17637218060e398a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The encoding for document 1  is :  [13, 29, 22, 49, 34, 10]\n",
      "The encoding for document 2  is :  [39, 36, 49, 34, 10, 20, 22, 49, 29]\n",
      "The encoding for document 3  is :  [12, 31, 29, 24, 45, 10, 19, 27, 36, 29, 10, 45]\n"
     ]
    }
   ],
   "source": [
    "vocab_size=50 \n",
    "encod_corp=[]\n",
    "for i,doc in enumerate(corp):\n",
    "    encod_corp.append(one_hot(doc,50))\n",
    "    print(\"The encoding for document\",i+1,\" is : \",one_hot(doc,50))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e58da092bf54cc13ea8eedb3b930b5857dc94b40"
   },
   "source": [
    "#### PADDING THE DOCS (to make very doc of same length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7dcb24072c22e932b98dcb15f17372430cd715f6"
   },
   "source": [
    "**The Keras Embedding layer requires all individual documents to be of same length.**  Hence we wil pad the shorter documents with 0 for now. Therefore now in Keras Embedding layer the **'input_length'**  will be equal to the length  (ie no of words) of the document with maximum length or maximum number of words.\n",
    "\n",
    "To pad the shorter documents I am using **pad_sequences** functon from the Keras library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_uuid": "0a0328b18b81c70f0db2dfbb7200fcde3ed665a5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The maximum number of words in any document is :  12\n"
     ]
    }
   ],
   "source": [
    "# length of maximum document. will be nedded whenever create embeddings for the words\n",
    "maxlen=-1\n",
    "for doc in corp:\n",
    "    tokens=nltk.word_tokenize(doc)\n",
    "    if(maxlen<len(tokens)):\n",
    "        maxlen=len(tokens)\n",
    "print(\"The maximum number of words in any document is : \",maxlen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_uuid": "22c873edc5421b21eec6419948d0e82d465c6373"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No of padded documents:  3\n"
     ]
    }
   ],
   "source": [
    "# now to create embeddings all of our docs need to be of same length. hence we can pad the docs with zeros.\n",
    "pad_corp=pad_sequences(encod_corp,maxlen=maxlen,padding='post',value=0.0)\n",
    "print(\"No of padded documents: \",len(pad_corp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_uuid": "f3cfbb0c557cd741bde8678dbab207590ea57960"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The padded encoding for document 1  is :  [13 29 22 49 34 10  0  0  0  0  0  0]\n",
      "The padded encoding for document 2  is :  [39 36 49 34 10 20 22 49 29  0  0  0]\n",
      "The padded encoding for document 3  is :  [12 31 29 24 45 10 19 27 36 29 10 45]\n"
     ]
    }
   ],
   "source": [
    "for i,doc in enumerate(pad_corp):\n",
    "     print(\"The padded encoding for document\",i+1,\" is : \",doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "95e0fa1b14f37562d504ef5b093e7d3812a26868"
   },
   "source": [
    "#### ACTUALLY CREATING THE EMBEDDINGS using KERAS EMBEDDING LAYER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "bb781af84c8e51a23241415749cb2bd1834f9676"
   },
   "source": [
    "Now all the documents are of same length (after padding). And so now we are ready to create and use the embeddings.\n",
    "\n",
    "**I will embed the words into vectors of 8 dimensions.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_uuid": "bdbc0bd1f475e8f6026b24e99fe8b353d226978c"
   },
   "outputs": [],
   "source": [
    "# specifying the input shape\n",
    "input=Input(shape=(no_docs,maxlen),dtype='float64')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_uuid": "6a2d829400ece99acebacbba69845fd999142f54"
   },
   "outputs": [],
   "source": [
    "'''\n",
    "shape of input. \n",
    "each document has 12 element or words which is the value of our maxlen variable.\n",
    "\n",
    "'''\n",
    "word_input=Input(shape=(maxlen,),dtype='float64')  \n",
    "\n",
    "# creating the embedding\n",
    "word_embedding=Embedding(input_dim=vocab_size,output_dim=8,input_length=maxlen)(word_input)\n",
    "\n",
    "word_vec=Flatten()(word_embedding) # flatten\n",
    "embed_model =Model([word_input],word_vec) # combining all into a Keras model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c71ce6d1c9719ed627989d985168f65129f6d9c6"
   },
   "source": [
    "**PARAMETERS OF THE EMBEDDING LAYER --- **\n",
    "\n",
    "**'input_dim' = the vocab size that we will choose**. \n",
    "In other words it is the number of unique words in the vocab.\n",
    "\n",
    "**'output_dim'  = the number of dimensions we wish to embed into**. Each word will be represented by a vector of this much dimensions.\n",
    "\n",
    "**'input_length' = lenght of the maximum document**. which is stored in maxlen variable in our case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "_uuid": "68bc772020f0fbcefe2a67e9d0422678522c6868"
   },
   "outputs": [],
   "source": [
    "embed_model.compile(optimizer=keras.optimizers.Adam(lr=1e-3),loss='binary_crossentropy',metrics=['acc']) \n",
    "# compiling the model. parameters can be tuned as always."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "_uuid": "cb509558221ab7e1df170340ac035af9f029f0f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'tensorflow.python.framework.ops.Tensor'>\n",
      "Tensor(\"embedding_1/embedding_lookup/Identity_1:0\", shape=(None, 12, 8), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(type(word_embedding))\n",
    "print(word_embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "_uuid": "0215812fb65979237ade681faa50af16ed1f0ac3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 12)                0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 12, 8)             400       \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 96)                0         \n",
      "=================================================================\n",
      "Total params: 400\n",
      "Trainable params: 400\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(embed_model.summary()) # summary of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "_uuid": "59db72f4b3c25b426087731f16317e24bf219e76"
   },
   "outputs": [],
   "source": [
    "embeddings=embed_model.predict(pad_corp) # finally getting the embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "_uuid": "c7d5ce3b7afd01b2afc9ba6bd9beb5269cdb5d68"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of embeddings :  (3, 96)\n",
      "[[ 2.2178996e-02 -8.4844977e-04  5.0595514e-03 -1.5644442e-02\n",
      "   1.3148475e-02 -2.1111691e-02 -4.6953607e-02  1.6933132e-02\n",
      "  -1.9370759e-02 -3.6944319e-02  3.8881872e-02  3.6577333e-02\n",
      "  -1.6315721e-02  4.2205278e-02 -8.3308443e-03  4.1705500e-02\n",
      "  -7.5347051e-03  2.3108069e-02 -2.4566937e-02  2.9110324e-02\n",
      "   3.4486245e-02 -7.0220828e-03  1.1225618e-02 -4.4404279e-02\n",
      "   4.4594098e-02  4.1802470e-02  1.9291509e-02  1.0375597e-02\n",
      "   3.2623593e-02  4.2842541e-02  4.1719783e-02  2.6841711e-02\n",
      "  -4.3161776e-02 -1.2376916e-02  3.5929929e-02 -1.7572306e-02\n",
      "  -3.0960847e-02 -4.2403806e-02  8.1663355e-03  3.5919357e-02\n",
      "  -4.7829151e-03  2.1874461e-02  2.7669106e-02 -4.1286945e-02\n",
      "  -2.4988502e-04  3.8018156e-02 -5.2970536e-03 -4.0472373e-03\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      " [ 3.8387429e-02  2.4441008e-02 -2.8917527e-02 -4.4114243e-02\n",
      "  -4.2978477e-02  1.5636895e-02 -2.3408055e-02  3.9117370e-02\n",
      "   4.3259148e-02 -2.7149320e-02 -3.9550126e-02  4.7661364e-05\n",
      "   2.1339778e-02 -3.3523284e-02  1.6476478e-02 -3.8844191e-02\n",
      "   4.4594098e-02  4.1802470e-02  1.9291509e-02  1.0375597e-02\n",
      "   3.2623593e-02  4.2842541e-02  4.1719783e-02  2.6841711e-02\n",
      "  -4.3161776e-02 -1.2376916e-02  3.5929929e-02 -1.7572306e-02\n",
      "  -3.0960847e-02 -4.2403806e-02  8.1663355e-03  3.5919357e-02\n",
      "  -4.7829151e-03  2.1874461e-02  2.7669106e-02 -4.1286945e-02\n",
      "  -2.4988502e-04  3.8018156e-02 -5.2970536e-03 -4.0472373e-03\n",
      "   3.4022223e-02  4.0392164e-02  1.0736108e-02 -3.9506495e-02\n",
      "   1.3714433e-03  3.2692943e-02  2.5043115e-03  1.4599707e-02\n",
      "  -7.5347051e-03  2.3108069e-02 -2.4566937e-02  2.9110324e-02\n",
      "   3.4486245e-02 -7.0220828e-03  1.1225618e-02 -4.4404279e-02\n",
      "   4.4594098e-02  4.1802470e-02  1.9291509e-02  1.0375597e-02\n",
      "   3.2623593e-02  4.2842541e-02  4.1719783e-02  2.6841711e-02\n",
      "  -1.9370759e-02 -3.6944319e-02  3.8881872e-02  3.6577333e-02\n",
      "  -1.6315721e-02  4.2205278e-02 -8.3308443e-03  4.1705500e-02\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03\n",
      "  -1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "  -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      " [ 1.7931554e-02 -1.4035035e-02 -5.3192861e-03  2.9640626e-02\n",
      "   1.1506140e-02  3.8858715e-02 -3.7870780e-03 -1.3351895e-02\n",
      "   4.0859487e-02  3.6289368e-02 -2.2248721e-02 -3.8032234e-02\n",
      "  -1.9106830e-02 -6.4059347e-04 -3.7247252e-02 -1.2183093e-02\n",
      "  -1.9370759e-02 -3.6944319e-02  3.8881872e-02  3.6577333e-02\n",
      "  -1.6315721e-02  4.2205278e-02 -8.3308443e-03  4.1705500e-02\n",
      "  -2.9727593e-03 -3.4558643e-02  3.0725066e-02 -3.0402768e-02\n",
      "  -2.9689407e-02 -3.9222203e-02 -1.7973136e-02 -1.6551293e-02\n",
      "   3.6562491e-02  1.2994003e-02  3.6690328e-02 -3.5830844e-02\n",
      "  -3.4020089e-02  2.5358979e-02 -4.2587508e-02  1.0704659e-02\n",
      "  -4.7829151e-03  2.1874461e-02  2.7669106e-02 -4.1286945e-02\n",
      "  -2.4988502e-04  3.8018156e-02 -5.2970536e-03 -4.0472373e-03\n",
      "  -6.1602481e-03 -1.7565023e-02  3.4272838e-02  1.7900337e-02\n",
      "  -1.2600232e-02 -4.9991440e-02 -2.3400342e-02  2.7633931e-02\n",
      "  -9.7090714e-03  4.1469220e-02 -6.6412799e-03  2.1320213e-02\n",
      "   2.8907325e-02 -9.5282421e-03  1.8322323e-02  2.2870786e-03\n",
      "   4.3259148e-02 -2.7149320e-02 -3.9550126e-02  4.7661364e-05\n",
      "   2.1339778e-02 -3.3523284e-02  1.6476478e-02 -3.8844191e-02\n",
      "  -1.9370759e-02 -3.6944319e-02  3.8881872e-02  3.6577333e-02\n",
      "  -1.6315721e-02  4.2205278e-02 -8.3308443e-03  4.1705500e-02\n",
      "  -4.7829151e-03  2.1874461e-02  2.7669106e-02 -4.1286945e-02\n",
      "  -2.4988502e-04  3.8018156e-02 -5.2970536e-03 -4.0472373e-03\n",
      "   3.6562491e-02  1.2994003e-02  3.6690328e-02 -3.5830844e-02\n",
      "  -3.4020089e-02  2.5358979e-02 -4.2587508e-02  1.0704659e-02]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Shape of embeddings : \",embeddings.shape)\n",
    "print(embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "_uuid": "1ce0944cbfab1c714c74ccc06f8eb1a846411ca8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of embeddings :  (3, 12, 8)\n",
      "[[[ 2.2178996e-02 -8.4844977e-04  5.0595514e-03 -1.5644442e-02\n",
      "    1.3148475e-02 -2.1111691e-02 -4.6953607e-02  1.6933132e-02]\n",
      "  [-1.9370759e-02 -3.6944319e-02  3.8881872e-02  3.6577333e-02\n",
      "   -1.6315721e-02  4.2205278e-02 -8.3308443e-03  4.1705500e-02]\n",
      "  [-7.5347051e-03  2.3108069e-02 -2.4566937e-02  2.9110324e-02\n",
      "    3.4486245e-02 -7.0220828e-03  1.1225618e-02 -4.4404279e-02]\n",
      "  [ 4.4594098e-02  4.1802470e-02  1.9291509e-02  1.0375597e-02\n",
      "    3.2623593e-02  4.2842541e-02  4.1719783e-02  2.6841711e-02]\n",
      "  [-4.3161776e-02 -1.2376916e-02  3.5929929e-02 -1.7572306e-02\n",
      "   -3.0960847e-02 -4.2403806e-02  8.1663355e-03  3.5919357e-02]\n",
      "  [-4.7829151e-03  2.1874461e-02  2.7669106e-02 -4.1286945e-02\n",
      "   -2.4988502e-04  3.8018156e-02 -5.2970536e-03 -4.0472373e-03]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]]\n",
      "\n",
      " [[ 3.8387429e-02  2.4441008e-02 -2.8917527e-02 -4.4114243e-02\n",
      "   -4.2978477e-02  1.5636895e-02 -2.3408055e-02  3.9117370e-02]\n",
      "  [ 4.3259148e-02 -2.7149320e-02 -3.9550126e-02  4.7661364e-05\n",
      "    2.1339778e-02 -3.3523284e-02  1.6476478e-02 -3.8844191e-02]\n",
      "  [ 4.4594098e-02  4.1802470e-02  1.9291509e-02  1.0375597e-02\n",
      "    3.2623593e-02  4.2842541e-02  4.1719783e-02  2.6841711e-02]\n",
      "  [-4.3161776e-02 -1.2376916e-02  3.5929929e-02 -1.7572306e-02\n",
      "   -3.0960847e-02 -4.2403806e-02  8.1663355e-03  3.5919357e-02]\n",
      "  [-4.7829151e-03  2.1874461e-02  2.7669106e-02 -4.1286945e-02\n",
      "   -2.4988502e-04  3.8018156e-02 -5.2970536e-03 -4.0472373e-03]\n",
      "  [ 3.4022223e-02  4.0392164e-02  1.0736108e-02 -3.9506495e-02\n",
      "    1.3714433e-03  3.2692943e-02  2.5043115e-03  1.4599707e-02]\n",
      "  [-7.5347051e-03  2.3108069e-02 -2.4566937e-02  2.9110324e-02\n",
      "    3.4486245e-02 -7.0220828e-03  1.1225618e-02 -4.4404279e-02]\n",
      "  [ 4.4594098e-02  4.1802470e-02  1.9291509e-02  1.0375597e-02\n",
      "    3.2623593e-02  4.2842541e-02  4.1719783e-02  2.6841711e-02]\n",
      "  [-1.9370759e-02 -3.6944319e-02  3.8881872e-02  3.6577333e-02\n",
      "   -1.6315721e-02  4.2205278e-02 -8.3308443e-03  4.1705500e-02]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]\n",
      "  [-1.3843823e-02 -9.7693093e-03  2.9820051e-02 -4.8851348e-02\n",
      "   -3.9608669e-02 -4.3516647e-02 -2.3724068e-02  2.1602139e-03]]\n",
      "\n",
      " [[ 1.7931554e-02 -1.4035035e-02 -5.3192861e-03  2.9640626e-02\n",
      "    1.1506140e-02  3.8858715e-02 -3.7870780e-03 -1.3351895e-02]\n",
      "  [ 4.0859487e-02  3.6289368e-02 -2.2248721e-02 -3.8032234e-02\n",
      "   -1.9106830e-02 -6.4059347e-04 -3.7247252e-02 -1.2183093e-02]\n",
      "  [-1.9370759e-02 -3.6944319e-02  3.8881872e-02  3.6577333e-02\n",
      "   -1.6315721e-02  4.2205278e-02 -8.3308443e-03  4.1705500e-02]\n",
      "  [-2.9727593e-03 -3.4558643e-02  3.0725066e-02 -3.0402768e-02\n",
      "   -2.9689407e-02 -3.9222203e-02 -1.7973136e-02 -1.6551293e-02]\n",
      "  [ 3.6562491e-02  1.2994003e-02  3.6690328e-02 -3.5830844e-02\n",
      "   -3.4020089e-02  2.5358979e-02 -4.2587508e-02  1.0704659e-02]\n",
      "  [-4.7829151e-03  2.1874461e-02  2.7669106e-02 -4.1286945e-02\n",
      "   -2.4988502e-04  3.8018156e-02 -5.2970536e-03 -4.0472373e-03]\n",
      "  [-6.1602481e-03 -1.7565023e-02  3.4272838e-02  1.7900337e-02\n",
      "   -1.2600232e-02 -4.9991440e-02 -2.3400342e-02  2.7633931e-02]\n",
      "  [-9.7090714e-03  4.1469220e-02 -6.6412799e-03  2.1320213e-02\n",
      "    2.8907325e-02 -9.5282421e-03  1.8322323e-02  2.2870786e-03]\n",
      "  [ 4.3259148e-02 -2.7149320e-02 -3.9550126e-02  4.7661364e-05\n",
      "    2.1339778e-02 -3.3523284e-02  1.6476478e-02 -3.8844191e-02]\n",
      "  [-1.9370759e-02 -3.6944319e-02  3.8881872e-02  3.6577333e-02\n",
      "   -1.6315721e-02  4.2205278e-02 -8.3308443e-03  4.1705500e-02]\n",
      "  [-4.7829151e-03  2.1874461e-02  2.7669106e-02 -4.1286945e-02\n",
      "   -2.4988502e-04  3.8018156e-02 -5.2970536e-03 -4.0472373e-03]\n",
      "  [ 3.6562491e-02  1.2994003e-02  3.6690328e-02 -3.5830844e-02\n",
      "   -3.4020089e-02  2.5358979e-02 -4.2587508e-02  1.0704659e-02]]]\n"
     ]
    }
   ],
   "source": [
    "embeddings=embeddings.reshape(-1,maxlen,8)\n",
    "print(\"Shape of embeddings : \",embeddings.shape) \n",
    "print(embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7ebe56daafe993cd6b6845231928ec237831707c"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d7cd46a4aaa6d7d1f43a81202add33c5b7f3b528"
   },
   "source": [
    "The resulting shape is (3,12,8).\n",
    "\n",
    "**3---> no of documents**\n",
    "\n",
    "**12---> each document is made of 12 words which was our maximum length of any document.**\n",
    "\n",
    "**& 8---> each word is 8 dimensional.**\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4312e187fb3604f9ba14f4ad5a46e2a44e7b49a9"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d8862c2ce0926f3fc06b813faa35d06ce72e87b6"
   },
   "source": [
    "#### GETTING ENCODING FOR A PARTICULAR WORD IN A SPECIFIC DOCUMENT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "_uuid": "6ac216640a7db94d9794a88256851d4e1e0c8858"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The encoding for  1 th word in 1 th document is : \n",
      "\n",
      " [ 0.022179   -0.00084845  0.00505955 -0.01564444  0.01314848 -0.02111169\n",
      " -0.04695361  0.01693313]\n",
      "The encoding for  2 th word in 1 th document is : \n",
      "\n",
      " [-0.01937076 -0.03694432  0.03888187  0.03657733 -0.01631572  0.04220528\n",
      " -0.00833084  0.0417055 ]\n",
      "The encoding for  3 th word in 1 th document is : \n",
      "\n",
      " [-0.00753471  0.02310807 -0.02456694  0.02911032  0.03448625 -0.00702208\n",
      "  0.01122562 -0.04440428]\n",
      "The encoding for  4 th word in 1 th document is : \n",
      "\n",
      " [0.0445941  0.04180247 0.01929151 0.0103756  0.03262359 0.04284254\n",
      " 0.04171978 0.02684171]\n",
      "The encoding for  5 th word in 1 th document is : \n",
      "\n",
      " [-0.04316178 -0.01237692  0.03592993 -0.01757231 -0.03096085 -0.04240381\n",
      "  0.00816634  0.03591936]\n",
      "The encoding for  6 th word in 1 th document is : \n",
      "\n",
      " [-0.00478292  0.02187446  0.02766911 -0.04128695 -0.00024989  0.03801816\n",
      " -0.00529705 -0.00404724]\n",
      "The encoding for  7 th word in 1 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  8 th word in 1 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  9 th word in 1 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  10 th word in 1 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  11 th word in 1 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  12 th word in 1 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  1 th word in 2 th document is : \n",
      "\n",
      " [ 0.03838743  0.02444101 -0.02891753 -0.04411424 -0.04297848  0.01563689\n",
      " -0.02340806  0.03911737]\n",
      "The encoding for  2 th word in 2 th document is : \n",
      "\n",
      " [ 4.3259148e-02 -2.7149320e-02 -3.9550126e-02  4.7661364e-05\n",
      "  2.1339778e-02 -3.3523284e-02  1.6476478e-02 -3.8844191e-02]\n",
      "The encoding for  3 th word in 2 th document is : \n",
      "\n",
      " [0.0445941  0.04180247 0.01929151 0.0103756  0.03262359 0.04284254\n",
      " 0.04171978 0.02684171]\n",
      "The encoding for  4 th word in 2 th document is : \n",
      "\n",
      " [-0.04316178 -0.01237692  0.03592993 -0.01757231 -0.03096085 -0.04240381\n",
      "  0.00816634  0.03591936]\n",
      "The encoding for  5 th word in 2 th document is : \n",
      "\n",
      " [-0.00478292  0.02187446  0.02766911 -0.04128695 -0.00024989  0.03801816\n",
      " -0.00529705 -0.00404724]\n",
      "The encoding for  6 th word in 2 th document is : \n",
      "\n",
      " [ 0.03402222  0.04039216  0.01073611 -0.03950649  0.00137144  0.03269294\n",
      "  0.00250431  0.01459971]\n",
      "The encoding for  7 th word in 2 th document is : \n",
      "\n",
      " [-0.00753471  0.02310807 -0.02456694  0.02911032  0.03448625 -0.00702208\n",
      "  0.01122562 -0.04440428]\n",
      "The encoding for  8 th word in 2 th document is : \n",
      "\n",
      " [0.0445941  0.04180247 0.01929151 0.0103756  0.03262359 0.04284254\n",
      " 0.04171978 0.02684171]\n",
      "The encoding for  9 th word in 2 th document is : \n",
      "\n",
      " [-0.01937076 -0.03694432  0.03888187  0.03657733 -0.01631572  0.04220528\n",
      " -0.00833084  0.0417055 ]\n",
      "The encoding for  10 th word in 2 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  11 th word in 2 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  12 th word in 2 th document is : \n",
      "\n",
      " [-0.01384382 -0.00976931  0.02982005 -0.04885135 -0.03960867 -0.04351665\n",
      " -0.02372407  0.00216021]\n",
      "The encoding for  1 th word in 3 th document is : \n",
      "\n",
      " [ 0.01793155 -0.01403503 -0.00531929  0.02964063  0.01150614  0.03885872\n",
      " -0.00378708 -0.01335189]\n",
      "The encoding for  2 th word in 3 th document is : \n",
      "\n",
      " [ 0.04085949  0.03628937 -0.02224872 -0.03803223 -0.01910683 -0.00064059\n",
      " -0.03724725 -0.01218309]\n",
      "The encoding for  3 th word in 3 th document is : \n",
      "\n",
      " [-0.01937076 -0.03694432  0.03888187  0.03657733 -0.01631572  0.04220528\n",
      " -0.00833084  0.0417055 ]\n",
      "The encoding for  4 th word in 3 th document is : \n",
      "\n",
      " [-0.00297276 -0.03455864  0.03072507 -0.03040277 -0.02968941 -0.0392222\n",
      " -0.01797314 -0.01655129]\n",
      "The encoding for  5 th word in 3 th document is : \n",
      "\n",
      " [ 0.03656249  0.012994    0.03669033 -0.03583084 -0.03402009  0.02535898\n",
      " -0.04258751  0.01070466]\n",
      "The encoding for  6 th word in 3 th document is : \n",
      "\n",
      " [-0.00478292  0.02187446  0.02766911 -0.04128695 -0.00024989  0.03801816\n",
      " -0.00529705 -0.00404724]\n",
      "The encoding for  7 th word in 3 th document is : \n",
      "\n",
      " [-0.00616025 -0.01756502  0.03427284  0.01790034 -0.01260023 -0.04999144\n",
      " -0.02340034  0.02763393]\n",
      "The encoding for  8 th word in 3 th document is : \n",
      "\n",
      " [-0.00970907  0.04146922 -0.00664128  0.02132021  0.02890733 -0.00952824\n",
      "  0.01832232  0.00228708]\n",
      "The encoding for  9 th word in 3 th document is : \n",
      "\n",
      " [ 4.3259148e-02 -2.7149320e-02 -3.9550126e-02  4.7661364e-05\n",
      "  2.1339778e-02 -3.3523284e-02  1.6476478e-02 -3.8844191e-02]\n",
      "The encoding for  10 th word in 3 th document is : \n",
      "\n",
      " [-0.01937076 -0.03694432  0.03888187  0.03657733 -0.01631572  0.04220528\n",
      " -0.00833084  0.0417055 ]\n",
      "The encoding for  11 th word in 3 th document is : \n",
      "\n",
      " [-0.00478292  0.02187446  0.02766911 -0.04128695 -0.00024989  0.03801816\n",
      " -0.00529705 -0.00404724]\n",
      "The encoding for  12 th word in 3 th document is : \n",
      "\n",
      " [ 0.03656249  0.012994    0.03669033 -0.03583084 -0.03402009  0.02535898\n",
      " -0.04258751  0.01070466]\n"
     ]
    }
   ],
   "source": [
    "for i,doc in enumerate(embeddings):\n",
    "    for j,word in enumerate(doc):\n",
    "        print(\"The encoding for \",j+1,\"th word\",\"in\",i+1,\"th document is : \\n\\n\",word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "19c665f54917a6aca7f65ea2ff7dbc47b96e90db"
   },
   "source": [
    "#### Now this makes it easier to visualize that we have 3(size of corp) documents with each consisting of 12(maxlen) words and each word mapped to a 8-dimensional vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f916e869a05cdf8f1e7e6030f7f7c108f64ddb23"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fdd3d2fb2fc58e889f78dc63ff31a5d55052c686"
   },
   "source": [
    "#### HOW TO WORK WITH A REAL PIECE OF TEXT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5d9d15978ec2de3cf27fd52d08ddfd1b756b1a83"
   },
   "source": [
    "Just like above we can now use any other document. We can sent_tokenize the doc into sentences.\n",
    "\n",
    "Each sentence has a list of words which we will integer encode using the 'one_hot' function as below. \n",
    "\n",
    "Now each sentence will be having different number of words. So we will need to pad the sequences to the sentence with maximum words.\n",
    "\n",
    "**At this point we are ready to feed the input to Keras Embedding layer as shown above.**\n",
    "\n",
    "**'input_dim' = the vocab size that we will choose**\n",
    "\n",
    "**'output_dim'  = the number of dimensions we wish to embed into**\n",
    "\n",
    "**'input_length' = lenght of the maximum document**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ed5997bd883965d864d75c86ae3bd050dd336efc"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8143b2d5460305133b30d1e8e0e70ee70839747f"
   },
   "source": [
    "## THE END !!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "49efc1cd691c5964f2efa1b46a757b84233dd176"
   },
   "source": [
    "**If you want to see the application of Keras embedding layer on a real task eg text classification then please check out my [this](https://github.com/mrc03/IMDB-Movie-Review-Sentiment-Analysis) repo on Github in which I have used the embeddings to perform sentiment analysis on IMdb movie review dataset.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "93078907fa0009405dbbcb046ca7cb65a8aa2af2"
   },
   "source": [
    "## [ Please Do upvote the kernel;) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "82790b65e6ccb380e3470ac3cb05b9793681d811"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}