8. Exploring Sentence and Domain Specific BERT/8.03. Exploring sentence-transformers library .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sr5a2PdXGS7a"
   },
   "source": [
    "# Exploring sentence-transformers library \n",
    "\n",
    "The sentence-transformers library can be installed using pip as shown in the following code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "id": "_GyedXY2GS7w"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install sentence_transformers==0.4.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TRE9MB2BGS70"
   },
   "source": [
    "\n",
    "The researchers of Sentence-BERT have also made their pre-trained Sentence-BERT models available online. All available pre-trained models can be found here (https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/).\n",
    "\n",
    "We can find pre-trained models named bert-base-nli-cls-token, bert-base-nli-mean-token, roberta-base-nli-max-tokens, distilbert-base-nli-mean-tokens, and so on. Let us understand what this means. \n",
    "\n",
    "- bert-base-nli-cls-token is a pre-trained Sentence-BERT model where we have taken a pre-trained BERT-base model and fine-tuned it with the NLI dataset and the model uses [CLS] token as the sentence representation. \n",
    "- bert-base-nli-mean-token is a pre-trained Sentence-BERT model where we have taken a pre-trained BERT-base model and fine-tuned it with the NLI dataset and the model uses mean pooling strategy for computing sentence representation.  \n",
    "- roberta-base-nli-max-tokens is a pre-trained Sentence-BERT model where we have taken a pre-trained RoBERTa-base model and fine-tuned it with the NLI dataset and the model uses max pooling strategy for computing sentence representation. \n",
    "- distilbert-base-nli-mean-token is a pre-trained Sentence-BERT model where we have taken a pre-trained DistilBERT-base model and fine-tuned it with the NLI dataset and the model uses mean pooling strategy for computing sentence representation. \n",
    "\n",
    "Thus, when we say the pre-trained Sentence-BERT model, it basically implies that we have taken a pre-trained BERT model and finetuned it using the Siamese/triplet network architecture. In the upcoming sections, we will learn how to use the pre-trained Sentence-BERT models. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Pj3LB7d5GS73"
   },
   "source": [
    "## Computing sentence representation using Sentence-BERT \n",
    "Let us see how to compute sentence representation using pre-trained Sentence-BERT. First, let us import the SentenceTransformer module from our sentence_transformers library. Use the following command:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "id": "trhb0DJ3GS75"
   },
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cjfZKirfGS76"
   },
   "source": [
    "Download and load the pre-trained Sentence-BERT:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "id": "wpySc5s1GS76"
   },
   "outputs": [],
   "source": [
    "model = SentenceTransformer('bert-base-nli-mean-tokens')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mRoNW25IGS78"
   },
   "source": [
    "\n",
    "Define the sentence to which we need to compute the sentence representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "id": "jJ8WS48sGS79"
   },
   "outputs": [],
   "source": [
    "sentence = 'paris is a beautiful city'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kMs1GQh5GS79"
   },
   "source": [
    "\n",
    "Compute the sentence representation using our pre-trained Sentence-BERT model with encode function: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "id": "kAo1vHAOGS7-"
   },
   "outputs": [],
   "source": [
    "sentence_representation = model.encode(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qIJlR_xAGS8A"
   },
   "source": [
    "\n",
    "Now, let us check the shape of our representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uhFm3udoGS8B",
    "outputId": "f86f79dd-c169-47a1-e154-09fbf4c7ffba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(768,)\n"
     ]
    }
   ],
   "source": [
    "print(sentence_representation.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YWCs20LpGS8C"
   },
   "source": [
    "\n",
    "As we can see our sentence representation size is 768. In this way, we can use the pre-trained Sentence-BERT model and obtain the fixed-length sentence representation. "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "8.03. Exploring sentence-transformers library .ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}