diffcse_evaluation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "diffcse_evaluation.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings\n",
        "\n",
        "To be published in [**NAACL 2022**](https://2022.naacl.org/)\n",
        "\n",
        "**TL;DR:** We proposed a new contrastive learning framework for learning unsupervised sentence embeddings that can achieve new state-of-the-art performance.\n",
        "<p align=\"center\"><img src=\"https://github.com/voidism/DiffCSE/raw/master/diffcse.png\" width=\"500\"></p>\n",
        "\n",
        "arXiv link: https://arxiv.org/abs/2204.10298  \n",
        "code link: https://github.com/voidism/DiffCSE  \n",
        "twitter discussion: https://twitter.com/YungSungChuang/status/1517518077902000129\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "gH2115o2tVsf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "1. git clone our repo\n",
        "2. install the customized transformers package (which supports a conditional transformer)\n",
        "3. install other requirements from pip"
      ],
      "metadata": {
        "id": "KWmCNnzduLyk"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yCxFW7_5shD-",
        "outputId": "2ad2ce94-0a7e-4df2-91cf-a5aa7966d85c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'DiffCSE'...\n",
            "remote: Enumerating objects: 1239, done.\u001b[K\n",
            "remote: Counting objects: 100% (1239/1239), done.\u001b[K\n",
            "remote: Compressing objects: 100% (900/900), done.\u001b[K\n",
            "remote: Total 1239 (delta 329), reused 1201 (delta 305), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (1239/1239), 6.34 MiB | 22.69 MiB/s, done.\n",
            "Resolving deltas: 100% (329/329), done.\n",
            "Processing /content/DiffCSE/transformers-4.2.1\n",
            "\u001b[33m  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.\n",
            "   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.\u001b[0m\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting sacremoses\n",
            "  Downloading sacremoses-0.0.49-py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 14.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.1) (4.64.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.1) (1.21.6)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.1) (4.11.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.1) (3.6.0)\n",
            "Collecting tokenizers==0.9.4\n",
            "  Downloading tokenizers-0.9.4-cp37-cp37m-manylinux2010_x86_64.whl (2.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.9 MB 57.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.1) (21.3)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.1) (2019.12.20)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.1) (2.23.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers==4.2.1) (4.1.1)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers==4.2.1) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==4.2.1) (3.0.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.2.1) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.2.1) (2021.10.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.2.1) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.2.1) (1.24.3)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.2.1) (7.1.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.2.1) (1.15.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.2.1) (1.1.0)\n",
            "Building wheels for collected packages: transformers\n",
            "  Building wheel for transformers (PEP 517) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for transformers: filename=transformers-4.2.1-py3-none-any.whl size=1735929 sha256=855a5a4fad0bc8144b7fd6e45b64277ff22f6f342dce62c4baebf21321c1d688\n",
            "  Stored in directory: /root/.cache/pip/wheels/9e/e2/11/6be874df3b4fbb09135be6e82423190f91e989d28f8bb3259c\n",
            "Successfully built transformers\n",
            "Installing collected packages: tokenizers, sacremoses, transformers\n",
            "Successfully installed sacremoses-0.0.49 tokenizers-0.9.4 transformers-4.2.1\n",
            "Collecting scipy==1.5.4\n",
            "  Downloading scipy-1.5.4-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 25.9 MB 1.3 MB/s \n",
            "\u001b[?25hCollecting datasets==1.2.1\n",
            "  Downloading datasets-1.2.1-py3-none-any.whl (159 kB)\n",
            "\u001b[K     |████████████████████████████████| 159 kB 71.0 MB/s \n",
            "\u001b[?25hCollecting pandas==1.1.5\n",
            "  Downloading pandas-1.1.5-cp37-cp37m-manylinux1_x86_64.whl (9.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 9.5 MB 49.1 MB/s \n",
            "\u001b[?25hCollecting scikit-learn==0.24.0\n",
            "  Downloading scikit_learn-0.24.0-cp37-cp37m-manylinux2010_x86_64.whl (22.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 22.3 MB 103.0 MB/s \n",
            "\u001b[?25hCollecting prettytable==2.1.0\n",
            "  Downloading prettytable-2.1.0-py3-none-any.whl (22 kB)\n",
            "Collecting gradio\n",
            "  Downloading gradio-2.9.4-py3-none-any.whl (2.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.9 MB 57.3 MB/s \n",
            "\u001b[?25hCollecting torch==1.7.1\n",
            "  Downloading torch-1.7.1-cp37-cp37m-manylinux1_x86_64.whl (776.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 776.8 MB 17 kB/s \n",
            "\u001b[?25hCollecting setuptools==49.3.0\n",
            "  Downloading setuptools-49.3.0-py3-none-any.whl (790 kB)\n",
            "\u001b[K     |████████████████████████████████| 790 kB 65.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.14.5 in /usr/local/lib/python3.7/dist-packages (from scipy==1.5.4->-r requirements.txt (line 1)) (1.21.6)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from datasets==1.2.1->-r requirements.txt (line 2)) (4.11.3)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from datasets==1.2.1->-r requirements.txt (line 2)) (2.23.0)\n",
            "Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets==1.2.1->-r requirements.txt (line 2)) (0.3.4)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from datasets==1.2.1->-r requirements.txt (line 2)) (0.70.12.2)\n",
            "Requirement already satisfied: pyarrow>=0.17.1 in /usr/local/lib/python3.7/dist-packages (from datasets==1.2.1->-r requirements.txt (line 2)) (6.0.1)\n",
            "Collecting tqdm<4.50.0,>=4.27\n",
            "  Downloading tqdm-4.49.0-py2.py3-none-any.whl (69 kB)\n",
            "\u001b[K     |████████████████████████████████| 69 kB 9.4 MB/s \n",
            "\u001b[?25hCollecting xxhash\n",
            "  Downloading xxhash-3.0.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
            "\u001b[K     |████████████████████████████████| 212 kB 76.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas==1.1.5->-r requirements.txt (line 3)) (2022.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas==1.1.5->-r requirements.txt (line 3)) (2.8.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.24.0->-r requirements.txt (line 4)) (3.1.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.24.0->-r requirements.txt (line 4)) (1.1.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prettytable==2.1.0->-r requirements.txt (line 5)) (0.2.5)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.7.1->-r requirements.txt (line 7)) (4.1.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas==1.1.5->-r requirements.txt (line 3)) (1.15.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets==1.2.1->-r requirements.txt (line 2)) (2021.10.8)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets==1.2.1->-r requirements.txt (line 2)) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets==1.2.1->-r requirements.txt (line 2)) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets==1.2.1->-r requirements.txt (line 2)) (2.10)\n",
            "Collecting paramiko\n",
            "  Downloading paramiko-2.10.3-py2.py3-none-any.whl (211 kB)\n",
            "\u001b[K     |████████████████████████████████| 211 kB 78.3 MB/s \n",
            "\u001b[?25hCollecting analytics-python\n",
            "  Downloading analytics_python-1.4.0-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from gradio->-r requirements.txt (line 6)) (7.1.2)\n",
            "Collecting ffmpy\n",
            "  Downloading ffmpy-0.3.0.tar.gz (4.8 kB)\n",
            "Collecting pydub\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Collecting aiohttp\n",
            "  Downloading aiohttp-3.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 49.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: Jinja2 in /usr/local/lib/python3.7/dist-packages (from gradio->-r requirements.txt (line 6)) (2.11.3)\n",
            "Collecting markdown-it-py[linkify,plugins]\n",
            "  Downloading markdown_it_py-2.1.0-py3-none-any.whl (84 kB)\n",
            "\u001b[K     |████████████████████████████████| 84 kB 4.5 MB/s \n",
            "\u001b[?25hCollecting orjson\n",
            "  Downloading orjson-3.6.8-cp37-cp37m-manylinux_2_24_x86_64.whl (253 kB)\n",
            "\u001b[K     |████████████████████████████████| 253 kB 77.4 MB/s \n",
            "\u001b[?25hCollecting python-multipart\n",
            "  Downloading python-multipart-0.0.5.tar.gz (32 kB)\n",
            "Collecting uvicorn\n",
            "  Downloading uvicorn-0.17.6-py3-none-any.whl (53 kB)\n",
            "\u001b[K     |████████████████████████████████| 53 kB 2.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from gradio->-r requirements.txt (line 6)) (3.2.2)\n",
            "Collecting fastapi\n",
            "  Downloading fastapi-0.75.2-py3-none-any.whl (54 kB)\n",
            "\u001b[K     |████████████████████████████████| 54 kB 3.4 MB/s \n",
            "\u001b[?25hCollecting pycryptodome\n",
            "  Downloading pycryptodome-3.14.1-cp35-abi3-manylinux2010_x86_64.whl (2.0 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.0 MB 61.7 MB/s \n",
            "\u001b[?25hCollecting asynctest==0.13.0\n",
            "  Downloading asynctest-0.13.0-py3-none-any.whl (26 kB)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->gradio->-r requirements.txt (line 6)) (2.0.12)\n",
            "Collecting yarl<2.0,>=1.0\n",
            "  Downloading yarl-1.7.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)\n",
            "\u001b[K     |████████████████████████████████| 271 kB 75.3 MB/s \n",
            "\u001b[?25hCollecting frozenlist>=1.1.1\n",
            "  Downloading frozenlist-1.3.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)\n",
            "\u001b[K     |████████████████████████████████| 144 kB 73.2 MB/s \n",
            "\u001b[?25hCollecting aiosignal>=1.1.2\n",
            "  Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->gradio->-r requirements.txt (line 6)) (21.4.0)\n",
            "Collecting multidict<7.0,>=4.5\n",
            "  Downloading multidict-6.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (94 kB)\n",
            "\u001b[K     |████████████████████████████████| 94 kB 4.4 MB/s \n",
            "\u001b[?25hCollecting async-timeout<5.0,>=4.0.0a3\n",
            "  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
            "Collecting monotonic>=1.5\n",
            "  Downloading monotonic-1.6-py2.py3-none-any.whl (8.2 kB)\n",
            "Collecting backoff==1.10.0\n",
            "  Downloading backoff-1.10.0-py2.py3-none-any.whl (31 kB)\n",
            "Collecting starlette==0.17.1\n",
            "  Downloading starlette-0.17.1-py3-none-any.whl (58 kB)\n",
            "\u001b[K     |████████████████████████████████| 58 kB 7.7 MB/s \n",
            "\u001b[?25hCollecting pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2\n",
            "  Downloading pydantic-1.9.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 10.9 MB 54.8 MB/s \n",
            "\u001b[?25hCollecting anyio<4,>=3.0.0\n",
            "  Downloading anyio-3.5.0-py3-none-any.whl (79 kB)\n",
            "\u001b[K     |████████████████████████████████| 79 kB 9.4 MB/s \n",
            "\u001b[?25hCollecting sniffio>=1.1\n",
            "  Downloading sniffio-1.2.0-py3-none-any.whl (10 kB)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->datasets==1.2.1->-r requirements.txt (line 2)) (3.8.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2->gradio->-r requirements.txt (line 6)) (2.0.1)\n",
            "Collecting mdurl~=0.1\n",
            "  Downloading mdurl-0.1.1-py3-none-any.whl (10 kB)\n",
            "Collecting mdit-py-plugins\n",
            "  Downloading mdit_py_plugins-0.3.0-py3-none-any.whl (43 kB)\n",
            "\u001b[K     |████████████████████████████████| 43 kB 2.6 MB/s \n",
            "\u001b[?25hCollecting linkify-it-py~=1.0\n",
            "  Downloading linkify_it_py-1.0.3-py3-none-any.whl (19 kB)\n",
            "Collecting uc-micro-py\n",
            "  Downloading uc_micro_py-1.0.1-py3-none-any.whl (6.2 kB)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->gradio->-r requirements.txt (line 6)) (3.0.8)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->gradio->-r requirements.txt (line 6)) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->gradio->-r requirements.txt (line 6)) (1.4.2)\n",
            "Collecting pynacl>=1.0.1\n",
            "  Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)\n",
            "\u001b[K     |████████████████████████████████| 856 kB 55.8 MB/s \n",
            "\u001b[?25hCollecting bcrypt>=3.1.3\n",
            "  Downloading bcrypt-3.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (61 kB)\n",
            "\u001b[K     |████████████████████████████████| 61 kB 551 kB/s \n",
            "\u001b[?25hCollecting cryptography>=2.5\n",
            "  Downloading cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.6 MB 52.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: cffi>=1.1 in /usr/local/lib/python3.7/dist-packages (from bcrypt>=3.1.3->paramiko->gradio->-r requirements.txt (line 6)) (1.15.0)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko->gradio->-r requirements.txt (line 6)) (2.21)\n",
            "Collecting h11>=0.8\n",
            "  Downloading h11-0.13.0-py3-none-any.whl (58 kB)\n",
            "\u001b[K     |████████████████████████████████| 58 kB 7.5 MB/s \n",
            "\u001b[?25hCollecting asgiref>=3.4.0\n",
            "  Downloading asgiref-3.5.0-py3-none-any.whl (22 kB)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.7/dist-packages (from uvicorn->gradio->-r requirements.txt (line 6)) (7.1.2)\n",
            "Building wheels for collected packages: ffmpy, python-multipart\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.0-py3-none-any.whl size=4712 sha256=82c386c5596de8d1d8f0dbeb57cf38b0225317b01bef21fc89dca6c74e53d465\n",
            "  Stored in directory: /root/.cache/pip/wheels/13/e4/6c/e8059816e86796a597c6e6b0d4c880630f51a1fcfa0befd5e6\n",
            "  Building wheel for python-multipart (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-multipart: filename=python_multipart-0.0.5-py3-none-any.whl size=31678 sha256=cefabddee3ac2c99c87bd9b97e7e0127892d7a7e0f607034b1c5476809390a2d\n",
            "  Stored in directory: /root/.cache/pip/wheels/2c/41/7c/bfd1c180534ffdcc0972f78c5758f89881602175d48a8bcd2c\n",
            "Successfully built ffmpy python-multipart\n",
            "Installing collected packages: sniffio, mdurl, uc-micro-py, multidict, markdown-it-py, frozenlist, anyio, yarl, starlette, pynacl, pydantic, monotonic, mdit-py-plugins, linkify-it-py, h11, cryptography, bcrypt, backoff, asynctest, async-timeout, asgiref, aiosignal, xxhash, uvicorn, tqdm, scipy, python-multipart, pydub, pycryptodome, paramiko, pandas, orjson, ffmpy, fastapi, analytics-python, aiohttp, torch, setuptools, scikit-learn, prettytable, gradio, datasets\n",
            "  Attempting uninstall: tqdm\n",
            "    Found existing installation: tqdm 4.64.0\n",
            "    Uninstalling tqdm-4.64.0:\n",
            "      Successfully uninstalled tqdm-4.64.0\n",
            "  Attempting uninstall: scipy\n",
            "    Found existing installation: scipy 1.4.1\n",
            "    Uninstalling scipy-1.4.1:\n",
            "      Successfully uninstalled scipy-1.4.1\n",
            "  Attempting uninstall: pandas\n",
            "    Found existing installation: pandas 1.3.5\n",
            "    Uninstalling pandas-1.3.5:\n",
            "      Successfully uninstalled pandas-1.3.5\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 1.10.0+cu111\n",
            "    Uninstalling torch-1.10.0+cu111:\n",
            "      Successfully uninstalled torch-1.10.0+cu111\n",
            "  Attempting uninstall: setuptools\n",
            "    Found existing installation: setuptools 57.4.0\n",
            "    Uninstalling setuptools-57.4.0:\n",
            "      Successfully uninstalled setuptools-57.4.0\n",
            "  Attempting uninstall: scikit-learn\n",
            "    Found existing installation: scikit-learn 1.0.2\n",
            "    Uninstalling scikit-learn-1.0.2:\n",
            "      Successfully uninstalled scikit-learn-1.0.2\n",
            "  Attempting uninstall: prettytable\n",
            "    Found existing installation: prettytable 3.2.0\n",
            "    Uninstalling prettytable-3.2.0:\n",
            "      Successfully uninstalled prettytable-3.2.0\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "tensorflow 2.8.0 requires tf-estimator-nightly==2.8.0.dev2021122109, which is not installed.\n",
            "yellowbrick 1.4 requires scikit-learn>=1.0.0, but you have scikit-learn 0.24.0 which is incompatible.\n",
            "torchvision 0.11.1+cu111 requires torch==1.10.0, but you have torch 1.7.1 which is incompatible.\n",
            "torchtext 0.11.0 requires torch==1.10.0, but you have torch 1.7.1 which is incompatible.\n",
            "torchaudio 0.10.0+cu111 requires torch==1.10.0, but you have torch 1.7.1 which is incompatible.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\n",
            "albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.\u001b[0m\n",
            "Successfully installed aiohttp-3.8.1 aiosignal-1.2.0 analytics-python-1.4.0 anyio-3.5.0 asgiref-3.5.0 async-timeout-4.0.2 asynctest-0.13.0 backoff-1.10.0 bcrypt-3.2.0 cryptography-36.0.2 datasets-1.2.1 fastapi-0.75.2 ffmpy-0.3.0 frozenlist-1.3.0 gradio-2.9.4 h11-0.13.0 linkify-it-py-1.0.3 markdown-it-py-2.1.0 mdit-py-plugins-0.3.0 mdurl-0.1.1 monotonic-1.6 multidict-6.0.2 orjson-3.6.8 pandas-1.1.5 paramiko-2.10.3 prettytable-2.1.0 pycryptodome-3.14.1 pydantic-1.9.0 pydub-0.25.1 pynacl-1.5.0 python-multipart-0.0.5 scikit-learn-0.24.0 scipy-1.5.4 setuptools-49.3.0 sniffio-1.2.0 starlette-0.17.1 torch-1.7.1 tqdm-4.49.0 uc-micro-py-1.0.1 uvicorn-0.17.6 xxhash-3.0.0 yarl-1.7.2\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/voidism/DiffCSE.git\n",
        "!cd DiffCSE/transformers-4.2.1 && pip install .\n",
        "!cd DiffCSE && pip install -r requirements.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Download dataset for SentEval"
      ],
      "metadata": {
        "id": "h9ZzKviDt88b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cd DiffCSE/SentEval/data/downstream/ && bash download_dataset.sh"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UCuC-N-et5ar",
        "outputId": "6a295068-8a5a-416e-f8e0-3ab9255bcd2a"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-04-23 18:11:09--  https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse/resolve/main/senteval.tar\n",
            "Resolving huggingface.co (huggingface.co)... 34.198.1.82, 34.224.55.150, 34.197.58.156, ...\n",
            "Connecting to huggingface.co (huggingface.co)|34.198.1.82|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://cdn-lfs.huggingface.co/datasets/princeton-nlp/datasets-for-simcse/bc43c148f7be97471c78fc4255399d3158cb99dfe8f2221999c918338b138c38 [following]\n",
            "--2022-04-23 18:11:09--  https://cdn-lfs.huggingface.co/datasets/princeton-nlp/datasets-for-simcse/bc43c148f7be97471c78fc4255399d3158cb99dfe8f2221999c918338b138c38\n",
            "Resolving cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)... 13.227.219.41, 13.227.219.2, 13.227.219.4, ...\n",
            "Connecting to cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)|13.227.219.41|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 89825280 (86M) [application/octet-stream]\n",
            "Saving to: ‘senteval.tar’\n",
            "\n",
            "senteval.tar        100%[===================>]  85.66M   108MB/s    in 0.8s    \n",
            "\n",
            "2022-04-23 18:11:10 (108 MB/s) - ‘senteval.tar’ saved [89825280/89825280]\n",
            "\n",
            "CR/\n",
            "CR/custrev.neg\n",
            "CR/custrev.pos\n",
            "MPQA/\n",
            "MPQA/mpqa.neg\n",
            "MPQA/mpqa.pos\n",
            "MR/\n",
            "MR/rt-polarity.neg\n",
            "MR/rt-polarity.pos\n",
            "MRPC/\n",
            "MRPC/msr_paraphrase_train.txt\n",
            "MRPC/msr_paraphrase_test.txt\n",
            "SICK/\n",
            "SICK/SICK_trial.txt\n",
            "SICK/SICK_train.txt\n",
            "SICK/SICK_test_annotated.txt\n",
            "SNLI/\n",
            "SNLI/s2.test\n",
            "SNLI/s1.train\n",
            "SNLI/s2.train\n",
            "SNLI/labels.dev\n",
            "SNLI/s1.test\n",
            "SNLI/labels.test\n",
            "SNLI/s2.dev\n",
            "SNLI/s1.dev\n",
            "SNLI/labels.train\n",
            "SST/\n",
            "SST/fine/\n",
            "SST/fine/sentiment-test\n",
            "SST/fine/sentiment-train\n",
            "SST/fine/sentiment-dev\n",
            "SST/binary/\n",
            "SST/binary/sentiment-test\n",
            "SST/binary/sentiment-train\n",
            "SST/binary/sentiment-dev\n",
            "STS/\n",
            "STS/STS12-en-test/\n",
            "STS/STS12-en-test/STS.gs.surprise.OnWN.txt\n",
            "STS/STS12-en-test/STS.input.surprise.OnWN.txt\n",
            "STS/STS12-en-test/STS.input.MSRpar.txt\n",
            "STS/STS12-en-test/STS.gs.ALL.txt\n",
            "STS/STS12-en-test/00-readme.txt\n",
            "STS/STS12-en-test/STS.gs.MSRvid.txt\n",
            "STS/STS12-en-test/STS.input.MSRvid.txt\n",
            "STS/STS12-en-test/STS.gs.MSRpar.txt\n",
            "STS/STS12-en-test/STS.input.surprise.SMTnews.txt\n",
            "STS/STS12-en-test/STS.gs.SMTeuroparl.txt\n",
            "STS/STS12-en-test/STS.gs.surprise.SMTnews.txt\n",
            "STS/STS12-en-test/STS.input.SMTeuroparl.txt\n",
            "STS/STS14-en-test/\n",
            "STS/STS14-en-test/STS.input.headlines.txt\n",
            "STS/STS14-en-test/STS.gs.OnWN.txt\n",
            "STS/STS14-en-test/STS.gs.images.txt\n",
            "STS/STS14-en-test/STS.gs.deft-news.txt\n",
            "STS/STS14-en-test/STS.gs.tweet-news.txt\n",
            "STS/STS14-en-test/sts2012-train.tgz\n",
            "STS/STS14-en-test/sts2013-test.tgz\n",
            "STS/STS14-en-test/00-readme.txt\n",
            "STS/STS14-en-test/STS.input.OnWN.txt\n",
            "STS/STS14-en-test/STS.input.deft-news.txt\n",
            "STS/STS14-en-test/sts2012-test.tgz\n",
            "STS/STS14-en-test/STS.input.deft-forum.txt\n",
            "STS/STS14-en-test/STS.output.headlines.txt\n",
            "STS/STS14-en-test/correlation-noconfidence.pl\n",
            "STS/STS14-en-test/STS.gs.headlines.txt\n",
            "STS/STS14-en-test/STS.gs.deft-forum.txt\n",
            "STS/STS14-en-test/STS.input.tweet-news.txt\n",
            "STS/STS14-en-test/STS.input.images.txt\n",
            "STS/STS15-en-test/\n",
            "STS/STS15-en-test/STS.input.headlines.txt\n",
            "STS/STS15-en-test/STS.gs.images.txt\n",
            "STS/STS15-en-test/STS.gs.answers-students.txt\n",
            "STS/STS15-en-test/00-readme.txt\n",
            "STS/STS15-en-test/STS.input.answers-students.txt\n",
            "STS/STS15-en-test/STS.input.answers-forums.LICENSE\n",
            "STS/STS15-en-test/STS.gs.answers-forums.txt\n",
            "STS/STS15-en-test/STS.input.answers-forums.txt\n",
            "STS/STS15-en-test/STS.gs.belief.txt\n",
            "STS/STS15-en-test/correlation-noconfidence.pl\n",
            "STS/STS15-en-test/STS.input.belief.txt\n",
            "STS/STS15-en-test/STS.gs.headlines.txt\n",
            "STS/STS15-en-test/STS.answers-forums.zip\n",
            "STS/STS15-en-test/corebaseline-tokencos.tar.gz\n",
            "STS/STS15-en-test/STS.input.images.txt\n",
            "STS/STS13-en-test/\n",
            "STS/STS13-en-test/STS.input.headlines.txt\n",
            "STS/STS13-en-test/STS.gs.OnWN.txt\n",
            "STS/STS13-en-test/correlation.pl\n",
            "STS/STS13-en-test/STS.input.FNWN.txt\n",
            "STS/STS13-en-test/STS.gs.FNWN.txt\n",
            "STS/STS13-en-test/STS.output.FNWN.txt\n",
            "STS/STS13-en-test/00-readme.txt\n",
            "STS/STS13-en-test/correlation-all.pl\n",
            "STS/STS13-en-test/STS.input.OnWN.txt\n",
            "STS/STS13-en-test/STS.output.headlines.txt\n",
            "STS/STS13-en-test/correct-output.pl\n",
            "STS/STS13-en-test/STS.gs.headlines.txt\n",
            "STS/STS13-en-test/STS.output.SMT.txt\n",
            "STS/STS13-en-test/STS.gs.SMT.txt\n",
            "STS/STS13-en-test/STS.output.OnWN.txt\n",
            "STS/STSBenchmark/\n",
            "STS/STSBenchmark/correlation.pl\n",
            "STS/STSBenchmark/sts-test.csv\n",
            "STS/STSBenchmark/readme.txt\n",
            "STS/STSBenchmark/LICENSE.txt\n",
            "STS/STSBenchmark/sts-train.csv\n",
            "STS/STSBenchmark/sts-dev.csv\n",
            "STS/STS16-en-test/\n",
            "STS/STS16-en-test/STS.input.headlines.txt\n",
            "STS/STS16-en-test/STS.gs.plagiarism.txt\n",
            "STS/STS16-en-test/STS.gs.question-question.txt\n",
            "STS/STS16-en-test/STS.input.question-question.txt\n",
            "STS/STS16-en-test/STS2016.input.headlines.ascii\n",
            "STS/STS16-en-test/README.txt\n",
            "STS/STS16-en-test/STS.gs.answer-answer.txt\n",
            "STS/STS16-en-test/STS2016.input.question-question.ascii\n",
            "STS/STS16-en-test/STS.input.postediting.txt\n",
            "STS/STS16-en-test/STS.input.plagiarism.txt\n",
            "STS/STS16-en-test/STS.gs.postediting.txt\n",
            "STS/STS16-en-test/LICENSE.txt\n",
            "STS/STS16-en-test/STS.input.answer-answer.txt\n",
            "STS/STS16-en-test/correlation-noconfidence.pl\n",
            "STS/STS16-en-test/STS.gs.headlines.txt\n",
            "STS/STS16-en-test/STS2016.input.answer-answer.ascii\n",
            "STS/STS16-en-test/STS2016.input.postediting.ascii\n",
            "STS/STS16-en-test/STS2016.input.plagiarism.ascii\n",
            "SUBJ/\n",
            "SUBJ/subj.subjective\n",
            "SUBJ/subj.objective\n",
            "TREC/\n",
            "TREC/train_5500.label\n",
            "TREC/TREC_10.label\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluate STS for BERT/RoBERTa\n",
        "\n",
        "Running STS evaluation for BERT (`voidism/diffcse-bert-base-uncased-sts`) and RoBERTa (`voidism/diffcse-roberta-base-sts`).\n",
        "\n",
        "You should get the results on Table 1 in our paper: \n",
        "* DiffCSE-BERT: Avg. 78.49\n",
        "* DiffCSE-RoBERTa: Avg. 78.21"
      ],
      "metadata": {
        "id": "SbfSvSE1ur6f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cd DiffCSE && python evaluation.py --model_name_or_path voidism/diffcse-bert-base-uncased-sts --pooler cls_before_pooler --task_set sts --mode test"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ICreOZ2PuTpY",
        "outputId": "eb52fdae-b5f7-498f-cae5-ed4ca1984899"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-04-23 04:49:27,466 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:49:27,609 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-sts/resolve/main/config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 04:49:27,611 : Attempting to acquire lock 139670952816464 on /root/.cache/huggingface/transformers/bb410801fc2bad2f3d4fc2cbe690d91f74b0e62555df9b75286267ad2df73d57.120b54d5f2a19d9e70f1038fcf9a1712f39d2f0bf49db2092f99abb21743a32c.lock\n",
            "2022-04-23 04:49:27,611 : Lock 139670952816464 acquired on /root/.cache/huggingface/transformers/bb410801fc2bad2f3d4fc2cbe690d91f74b0e62555df9b75286267ad2df73d57.120b54d5f2a19d9e70f1038fcf9a1712f39d2f0bf49db2092f99abb21743a32c.lock\n",
            "2022-04-23 04:49:27,612 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:49:27,755 : https://huggingface.co:443 \"GET /voidism/diffcse-bert-base-uncased-sts/resolve/main/config.json HTTP/1.1\" 200 619\n",
            "Downloading: 100% 619/619 [00:00<00:00, 438kB/s]\n",
            "2022-04-23 04:49:27,758 : Attempting to release lock 139670952816464 on /root/.cache/huggingface/transformers/bb410801fc2bad2f3d4fc2cbe690d91f74b0e62555df9b75286267ad2df73d57.120b54d5f2a19d9e70f1038fcf9a1712f39d2f0bf49db2092f99abb21743a32c.lock\n",
            "2022-04-23 04:49:27,759 : Lock 139670952816464 released on /root/.cache/huggingface/transformers/bb410801fc2bad2f3d4fc2cbe690d91f74b0e62555df9b75286267ad2df73d57.120b54d5f2a19d9e70f1038fcf9a1712f39d2f0bf49db2092f99abb21743a32c.lock\n",
            "2022-04-23 04:49:27,761 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:49:27,904 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-sts/resolve/main/pytorch_model.bin HTTP/1.1\" 302 0\n",
            "2022-04-23 04:49:27,907 : Attempting to acquire lock 139670970255696 on /root/.cache/huggingface/transformers/2d74798d1d601c658af832988697030af0194d841781f7a217021f9b4ff78a95.f42a38d568fb3df11880e7c9c69d6be71067e33bd6aad72f92cc5aa2c2fc8fe7.lock\n",
            "2022-04-23 04:49:27,907 : Lock 139670970255696 acquired on /root/.cache/huggingface/transformers/2d74798d1d601c658af832988697030af0194d841781f7a217021f9b4ff78a95.f42a38d568fb3df11880e7c9c69d6be71067e33bd6aad72f92cc5aa2c2fc8fe7.lock\n",
            "2022-04-23 04:49:27,908 : Starting new HTTPS connection (1): cdn-lfs.huggingface.co:443\n",
            "2022-04-23 04:49:28,128 : https://cdn-lfs.huggingface.co:443 \"GET /repos/f7/72/f772365a6efccdb9dfc3d8172e07c9776c902440860dafbdbc701e9902cef12f/39f53fc7e1db892f6a68aea4d1566c0e09a93bfaaa501380a528ad418d7e2a8c HTTP/1.1\" 200 1245026652\n",
            "Downloading: 100% 1.25G/1.25G [00:30<00:00, 41.4MB/s]\n",
            "2022-04-23 04:49:58,172 : Attempting to release lock 139670970255696 on /root/.cache/huggingface/transformers/2d74798d1d601c658af832988697030af0194d841781f7a217021f9b4ff78a95.f42a38d568fb3df11880e7c9c69d6be71067e33bd6aad72f92cc5aa2c2fc8fe7.lock\n",
            "2022-04-23 04:49:58,172 : Lock 139670970255696 released on /root/.cache/huggingface/transformers/2d74798d1d601c658af832988697030af0194d841781f7a217021f9b4ff78a95.f42a38d568fb3df11880e7c9c69d6be71067e33bd6aad72f92cc5aa2c2fc8fe7.lock\n",
            "Some weights of BertModel were not initialized from the model checkpoint at voidism/diffcse-bert-base-uncased-sts and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
            "2022-04-23 04:50:01,731 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:50:01,884 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-sts/resolve/main/config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 04:50:01,890 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:50:02,027 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-sts/resolve/main/vocab.txt HTTP/1.1\" 200 0\n",
            "2022-04-23 04:50:02,029 : Attempting to acquire lock 139670951999888 on /root/.cache/huggingface/transformers/34e542428de58826881c04d0aaae600d28a9b618c763afb155990ca4aa05dffd.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock\n",
            "2022-04-23 04:50:02,029 : Lock 139670951999888 acquired on /root/.cache/huggingface/transformers/34e542428de58826881c04d0aaae600d28a9b618c763afb155990ca4aa05dffd.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock\n",
            "2022-04-23 04:50:02,031 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:50:02,179 : https://huggingface.co:443 \"GET /voidism/diffcse-bert-base-uncased-sts/resolve/main/vocab.txt HTTP/1.1\" 200 231508\n",
            "Downloading: 100% 232k/232k [00:00<00:00, 1.93MB/s]\n",
            "2022-04-23 04:50:02,301 : Attempting to release lock 139670951999888 on /root/.cache/huggingface/transformers/34e542428de58826881c04d0aaae600d28a9b618c763afb155990ca4aa05dffd.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock\n",
            "2022-04-23 04:50:02,301 : Lock 139670951999888 released on /root/.cache/huggingface/transformers/34e542428de58826881c04d0aaae600d28a9b618c763afb155990ca4aa05dffd.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock\n",
            "2022-04-23 04:50:02,303 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:50:02,435 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-sts/resolve/main/tokenizer.json HTTP/1.1\" 404 0\n",
            "2022-04-23 04:50:02,438 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:50:02,584 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-sts/resolve/main/added_tokens.json HTTP/1.1\" 404 0\n",
            "2022-04-23 04:50:02,588 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:50:02,728 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-sts/resolve/main/special_tokens_map.json HTTP/1.1\" 404 0\n",
            "2022-04-23 04:50:02,733 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:50:02,882 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-sts/resolve/main/tokenizer_config.json HTTP/1.1\" 404 0\n",
            "2022-04-23 04:50:05,239 : ***** Transfer task : STS12 *****\n",
            "\n",
            "\n",
            "./SentEval/senteval/sts.py:42: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
            "  sent1 = np.array([s.split() for s in sent1])[not_empty_idx]\n",
            "./SentEval/senteval/sts.py:43: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
            "  sent2 = np.array([s.split() for s in sent2])[not_empty_idx]\n",
            "2022-04-23 04:50:13,518 : MSRpar : pearson = 0.6389, spearman = 0.6455\n",
            "2022-04-23 04:50:16,955 : MSRvid : pearson = 0.8708, spearman = 0.8768\n",
            "2022-04-23 04:50:19,971 : SMTeuroparl : pearson = 0.4632, spearman = 0.6234\n",
            "2022-04-23 04:50:25,592 : surprise.OnWN : pearson = 0.7760, spearman = 0.7271\n",
            "2022-04-23 04:50:28,489 : surprise.SMTnews : pearson = 0.7100, spearman = 0.6054\n",
            "2022-04-23 04:50:28,492 : ALL : Pearson = 0.7942,             Spearman = 0.7228\n",
            "2022-04-23 04:50:28,492 : ALL (weighted average) : Pearson = 0.7111,             Spearman = 0.7126\n",
            "2022-04-23 04:50:28,492 : ALL (average) : Pearson = 0.6918,             Spearman = 0.6956\n",
            "\n",
            "2022-04-23 04:50:28,495 : ***** Transfer task : STS13 (-SMT) *****\n",
            "\n",
            "\n",
            "2022-04-23 04:50:31,255 : FNWN : pearson = 0.6286, spearman = 0.6288\n",
            "2022-04-23 04:50:35,127 : headlines : pearson = 0.8251, spearman = 0.8403\n",
            "2022-04-23 04:50:38,227 : OnWN : pearson = 0.8688, spearman = 0.8456\n",
            "2022-04-23 04:50:38,229 : ALL : Pearson = 0.8354,             Spearman = 0.8443\n",
            "2022-04-23 04:50:38,229 : ALL (weighted average) : Pearson = 0.8167,             Spearman = 0.8156\n",
            "2022-04-23 04:50:38,229 : ALL (average) : Pearson = 0.7741,             Spearman = 0.7716\n",
            "\n",
            "2022-04-23 04:50:38,231 : ***** Transfer task : STS14 *****\n",
            "\n",
            "\n",
            "2022-04-23 04:50:41,182 : deft-forum : pearson = 0.5876, spearman = 0.5872\n",
            "2022-04-23 04:50:44,501 : deft-news : pearson = 0.8260, spearman = 0.7944\n",
            "2022-04-23 04:50:49,049 : headlines : pearson = 0.8089, spearman = 0.8032\n",
            "2022-04-23 04:50:53,378 : images : pearson = 0.8515, spearman = 0.8250\n",
            "2022-04-23 04:50:57,853 : OnWN : pearson = 0.8897, spearman = 0.8674\n",
            "2022-04-23 04:51:03,715 : tweet-news : pearson = 0.7950, spearman = 0.7308\n",
            "2022-04-23 04:51:03,718 : ALL : Pearson = 0.8009,             Spearman = 0.7647\n",
            "2022-04-23 04:51:03,719 : ALL (weighted average) : Pearson = 0.8056,             Spearman = 0.7793\n",
            "2022-04-23 04:51:03,719 : ALL (average) : Pearson = 0.7931,             Spearman = 0.7680\n",
            "\n",
            "2022-04-23 04:51:03,722 : ***** Transfer task : STS15 *****\n",
            "\n",
            "\n",
            "2022-04-23 04:51:07,689 : answers-forums : pearson = 0.7508, spearman = 0.7581\n",
            "2022-04-23 04:51:11,999 : answers-students : pearson = 0.7535, spearman = 0.7623\n",
            "2022-04-23 04:51:16,198 : belief : pearson = 0.8221, spearman = 0.8458\n",
            "2022-04-23 04:51:20,895 : headlines : pearson = 0.8362, spearman = 0.8480\n",
            "2022-04-23 04:51:25,355 : images : pearson = 0.8691, spearman = 0.8917\n",
            "2022-04-23 04:51:25,358 : ALL : Pearson = 0.8269,             Spearman = 0.8390\n",
            "2022-04-23 04:51:25,358 : ALL (weighted average) : Pearson = 0.8113,             Spearman = 0.8260\n",
            "2022-04-23 04:51:25,358 : ALL (average) : Pearson = 0.8063,             Spearman = 0.8212\n",
            "\n",
            "2022-04-23 04:51:25,363 : ***** Transfer task : STS16 *****\n",
            "\n",
            "\n",
            "2022-04-23 04:51:27,273 : answer-answer : pearson = 0.7056, spearman = 0.7098\n",
            "2022-04-23 04:51:28,640 : headlines : pearson = 0.8104, spearman = 0.8385\n",
            "2022-04-23 04:51:30,444 : plagiarism : pearson = 0.8494, spearman = 0.8635\n",
            "2022-04-23 04:51:33,457 : postediting : pearson = 0.8609, spearman = 0.8860\n",
            "2022-04-23 04:51:34,754 : question-question : pearson = 0.7425, spearman = 0.7691\n",
            "2022-04-23 04:51:34,755 : ALL : Pearson = 0.7870,             Spearman = 0.8054\n",
            "2022-04-23 04:51:34,756 : ALL (weighted average) : Pearson = 0.7940,             Spearman = 0.8133\n",
            "2022-04-23 04:51:34,756 : ALL (average) : Pearson = 0.7938,             Spearman = 0.8134\n",
            "\n",
            "2022-04-23 04:51:34,757 : \n",
            "\n",
            "***** Transfer task : STSBenchmark*****\n",
            "\n",
            "\n",
            "2022-04-23 04:52:21,702 : train : pearson = 0.8221, spearman = 0.8094\n",
            "2022-04-23 04:52:35,399 : dev : pearson = 0.8378, spearman = 0.8456\n",
            "2022-04-23 04:52:46,885 : test : pearson = 0.8033, spearman = 0.8059\n",
            "2022-04-23 04:52:46,892 : ALL : Pearson = 0.8228,             Spearman = 0.8183\n",
            "2022-04-23 04:52:46,892 : ALL (weighted average) : Pearson = 0.8218,             Spearman = 0.8152\n",
            "2022-04-23 04:52:46,892 : ALL (average) : Pearson = 0.8211,             Spearman = 0.8203\n",
            "\n",
            "2022-04-23 04:52:46,898 : \n",
            "\n",
            "***** Transfer task : SICKRelatedness*****\n",
            "\n",
            "\n",
            "2022-04-23 04:53:16,383 : train : pearson = 0.8055, spearman = 0.7204\n",
            "2022-04-23 04:53:20,130 : dev : pearson = 0.8024, spearman = 0.7464\n",
            "2022-04-23 04:53:52,510 : test : pearson = 0.7973, spearman = 0.7123\n",
            "2022-04-23 04:53:52,517 : ALL : Pearson = 0.8013,             Spearman = 0.7177\n",
            "2022-04-23 04:53:52,517 : ALL (weighted average) : Pearson = 0.8013,             Spearman = 0.7177\n",
            "2022-04-23 04:53:52,517 : ALL (average) : Pearson = 0.8017,             Spearman = 0.7264\n",
            "\n",
            "------ test ------\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+-------+\n",
            "| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness |  Avg. |\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+-------+\n",
            "| 72.28 | 84.43 | 76.47 | 83.90 | 80.54 |    80.59     |      71.23      | 78.49 |\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+-------+\n",
            "+------+------+------+------+------+------+------+------+\n",
            "|  MR  |  CR  | SUBJ | MPQA | SST2 | TREC | MRPC | Avg. |\n",
            "+------+------+------+------+------+------+------+------+\n",
            "| 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |\n",
            "+------+------+------+------+------+------+------+------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!cd DiffCSE && python evaluation.py --model_name_or_path voidism/diffcse-roberta-base-sts --pooler cls_before_pooler --task_set sts --mode test"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FcSud7dIu9mZ",
        "outputId": "437adec6-7d24-409f-86b6-1791f5d17386"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-04-23 04:53:55,211 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:53:55,404 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 04:53:55,406 : Attempting to acquire lock 139653564931984 on /root/.cache/huggingface/transformers/71d79e292d62b243ba0a6d87f56a5e5cfc30286f655a93ed3e13b486aa285ad2.da5da0a4fc0135920c6a291cf5c0fb324dfa4097155838d15ac6927c8acb961a.lock\n",
            "2022-04-23 04:53:55,406 : Lock 139653564931984 acquired on /root/.cache/huggingface/transformers/71d79e292d62b243ba0a6d87f56a5e5cfc30286f655a93ed3e13b486aa285ad2.da5da0a4fc0135920c6a291cf5c0fb324dfa4097155838d15ac6927c8acb961a.lock\n",
            "2022-04-23 04:53:55,408 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:53:55,551 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-sts/resolve/main/config.json HTTP/1.1\" 200 662\n",
            "Downloading: 100% 662/662 [00:00<00:00, 444kB/s]\n",
            "2022-04-23 04:53:55,555 : Attempting to release lock 139653564931984 on /root/.cache/huggingface/transformers/71d79e292d62b243ba0a6d87f56a5e5cfc30286f655a93ed3e13b486aa285ad2.da5da0a4fc0135920c6a291cf5c0fb324dfa4097155838d15ac6927c8acb961a.lock\n",
            "2022-04-23 04:53:55,555 : Lock 139653564931984 released on /root/.cache/huggingface/transformers/71d79e292d62b243ba0a6d87f56a5e5cfc30286f655a93ed3e13b486aa285ad2.da5da0a4fc0135920c6a291cf5c0fb324dfa4097155838d15ac6927c8acb961a.lock\n",
            "2022-04-23 04:53:55,558 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:53:55,703 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/pytorch_model.bin HTTP/1.1\" 302 0\n",
            "2022-04-23 04:53:55,706 : Attempting to acquire lock 139653564931984 on /root/.cache/huggingface/transformers/928f58ecafc64d45eab666e8b1964aecf59c36cb66e4a4e3ac4e7d32c505fa42.f98bbb5411ea99c70aea962f40302a825522815ea71bb9fa9c879ba56651ec26.lock\n",
            "2022-04-23 04:53:55,706 : Lock 139653564931984 acquired on /root/.cache/huggingface/transformers/928f58ecafc64d45eab666e8b1964aecf59c36cb66e4a4e3ac4e7d32c505fa42.f98bbb5411ea99c70aea962f40302a825522815ea71bb9fa9c879ba56651ec26.lock\n",
            "2022-04-23 04:53:55,708 : Starting new HTTPS connection (1): cdn-lfs.huggingface.co:443\n",
            "2022-04-23 04:53:55,964 : https://cdn-lfs.huggingface.co:443 \"GET /repos/d0/53/d053794ba49357d59758351b9dc9962e2b662b23a34dff9cf6eecebc8c75392d/b792aba0b664d672181d12edb092a4e53501e5197f331ff3c97ceed4fed45320 HTTP/1.1\" 200 1487809454\n",
            "Downloading: 100% 1.49G/1.49G [00:36<00:00, 41.0MB/s]\n",
            "2022-04-23 04:54:32,211 : Attempting to release lock 139653564931984 on /root/.cache/huggingface/transformers/928f58ecafc64d45eab666e8b1964aecf59c36cb66e4a4e3ac4e7d32c505fa42.f98bbb5411ea99c70aea962f40302a825522815ea71bb9fa9c879ba56651ec26.lock\n",
            "2022-04-23 04:54:32,211 : Lock 139653564931984 released on /root/.cache/huggingface/transformers/928f58ecafc64d45eab666e8b1964aecf59c36cb66e4a4e3ac4e7d32c505fa42.f98bbb5411ea99c70aea962f40302a825522815ea71bb9fa9c879ba56651ec26.lock\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at voidism/diffcse-roberta-base-sts and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
            "2022-04-23 04:54:36,501 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:36,669 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 04:54:36,673 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:36,812 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/vocab.json HTTP/1.1\" 200 0\n",
            "2022-04-23 04:54:36,815 : Attempting to acquire lock 139653564947920 on /root/.cache/huggingface/transformers/39ce30ee78e30343794f8f72802fbdb951441fc671aefcbba37a16397d3cb906.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7.lock\n",
            "2022-04-23 04:54:36,815 : Lock 139653564947920 acquired on /root/.cache/huggingface/transformers/39ce30ee78e30343794f8f72802fbdb951441fc671aefcbba37a16397d3cb906.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7.lock\n",
            "2022-04-23 04:54:36,817 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:36,969 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-sts/resolve/main/vocab.json HTTP/1.1\" 200 798293\n",
            "Downloading: 100% 798k/798k [00:00<00:00, 4.69MB/s]\n",
            "2022-04-23 04:54:37,140 : Attempting to release lock 139653564947920 on /root/.cache/huggingface/transformers/39ce30ee78e30343794f8f72802fbdb951441fc671aefcbba37a16397d3cb906.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7.lock\n",
            "2022-04-23 04:54:37,141 : Lock 139653564947920 released on /root/.cache/huggingface/transformers/39ce30ee78e30343794f8f72802fbdb951441fc671aefcbba37a16397d3cb906.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7.lock\n",
            "2022-04-23 04:54:37,142 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:37,278 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/merges.txt HTTP/1.1\" 200 0\n",
            "2022-04-23 04:54:37,280 : Attempting to acquire lock 139653548842832 on /root/.cache/huggingface/transformers/c67623cc1f90ca3eafb40031f303cbc07f89775a883236490cb102fba4ba705a.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435.lock\n",
            "2022-04-23 04:54:37,280 : Lock 139653548842832 acquired on /root/.cache/huggingface/transformers/c67623cc1f90ca3eafb40031f303cbc07f89775a883236490cb102fba4ba705a.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435.lock\n",
            "2022-04-23 04:54:37,281 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:37,425 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-sts/resolve/main/merges.txt HTTP/1.1\" 200 456356\n",
            "Downloading: 100% 456k/456k [00:00<00:00, 3.35MB/s]\n",
            "2022-04-23 04:54:37,563 : Attempting to release lock 139653548842832 on /root/.cache/huggingface/transformers/c67623cc1f90ca3eafb40031f303cbc07f89775a883236490cb102fba4ba705a.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435.lock\n",
            "2022-04-23 04:54:37,563 : Lock 139653548842832 released on /root/.cache/huggingface/transformers/c67623cc1f90ca3eafb40031f303cbc07f89775a883236490cb102fba4ba705a.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435.lock\n",
            "2022-04-23 04:54:37,565 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:37,707 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/tokenizer.json HTTP/1.1\" 404 0\n",
            "2022-04-23 04:54:37,710 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:37,850 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/added_tokens.json HTTP/1.1\" 404 0\n",
            "2022-04-23 04:54:37,853 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:38,001 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/special_tokens_map.json HTTP/1.1\" 200 0\n",
            "2022-04-23 04:54:38,003 : Attempting to acquire lock 139653548841552 on /root/.cache/huggingface/transformers/c81b6ac1dc7ab40d68699fdf5d8da8c565423fdf80292e787d51c53879058b7f.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342.lock\n",
            "2022-04-23 04:54:38,003 : Lock 139653548841552 acquired on /root/.cache/huggingface/transformers/c81b6ac1dc7ab40d68699fdf5d8da8c565423fdf80292e787d51c53879058b7f.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342.lock\n",
            "2022-04-23 04:54:38,005 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:38,150 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-sts/resolve/main/special_tokens_map.json HTTP/1.1\" 200 239\n",
            "Downloading: 100% 239/239 [00:00<00:00, 163kB/s]\n",
            "2022-04-23 04:54:38,154 : Attempting to release lock 139653548841552 on /root/.cache/huggingface/transformers/c81b6ac1dc7ab40d68699fdf5d8da8c565423fdf80292e787d51c53879058b7f.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342.lock\n",
            "2022-04-23 04:54:38,154 : Lock 139653548841552 released on /root/.cache/huggingface/transformers/c81b6ac1dc7ab40d68699fdf5d8da8c565423fdf80292e787d51c53879058b7f.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342.lock\n",
            "2022-04-23 04:54:38,156 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:38,294 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-sts/resolve/main/tokenizer_config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 04:54:38,295 : Attempting to acquire lock 139653548863888 on /root/.cache/huggingface/transformers/1e7730546b6543000f557e23c0447a6ef7730957b015b883c469aa5d226e10fb.3fd6ad28580f134ac5fac0b861423fd64502e7b6abeb15831037d02b89280db7.lock\n",
            "2022-04-23 04:54:38,296 : Lock 139653548863888 acquired on /root/.cache/huggingface/transformers/1e7730546b6543000f557e23c0447a6ef7730957b015b883c469aa5d226e10fb.3fd6ad28580f134ac5fac0b861423fd64502e7b6abeb15831037d02b89280db7.lock\n",
            "2022-04-23 04:54:38,297 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 04:54:38,432 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-sts/resolve/main/tokenizer_config.json HTTP/1.1\" 200 265\n",
            "Downloading: 100% 265/265 [00:00<00:00, 194kB/s]\n",
            "2022-04-23 04:54:38,435 : Attempting to release lock 139653548863888 on /root/.cache/huggingface/transformers/1e7730546b6543000f557e23c0447a6ef7730957b015b883c469aa5d226e10fb.3fd6ad28580f134ac5fac0b861423fd64502e7b6abeb15831037d02b89280db7.lock\n",
            "2022-04-23 04:54:38,435 : Lock 139653548863888 released on /root/.cache/huggingface/transformers/1e7730546b6543000f557e23c0447a6ef7730957b015b883c469aa5d226e10fb.3fd6ad28580f134ac5fac0b861423fd64502e7b6abeb15831037d02b89280db7.lock\n",
            "2022-04-23 04:54:41,506 : ***** Transfer task : STS12 *****\n",
            "\n",
            "\n",
            "./SentEval/senteval/sts.py:42: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
            "  sent1 = np.array([s.split() for s in sent1])[not_empty_idx]\n",
            "./SentEval/senteval/sts.py:43: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
            "  sent2 = np.array([s.split() for s in sent2])[not_empty_idx]\n",
            "2022-04-23 04:54:50,128 : MSRpar : pearson = 0.6426, spearman = 0.6354\n",
            "2022-04-23 04:54:53,493 : MSRvid : pearson = 0.8654, spearman = 0.8613\n",
            "2022-04-23 04:54:56,524 : SMTeuroparl : pearson = 0.5107, spearman = 0.5897\n",
            "2022-04-23 04:55:02,007 : surprise.OnWN : pearson = 0.7528, spearman = 0.7204\n",
            "2022-04-23 04:55:04,959 : surprise.SMTnews : pearson = 0.7140, spearman = 0.5953\n",
            "2022-04-23 04:55:04,962 : ALL : Pearson = 0.7730,             Spearman = 0.7005\n",
            "2022-04-23 04:55:04,962 : ALL (weighted average) : Pearson = 0.7126,             Spearman = 0.6985\n",
            "2022-04-23 04:55:04,962 : ALL (average) : Pearson = 0.6971,             Spearman = 0.6804\n",
            "\n",
            "2022-04-23 04:55:04,965 : ***** Transfer task : STS13 (-SMT) *****\n",
            "\n",
            "\n",
            "2022-04-23 04:55:07,776 : FNWN : pearson = 0.6278, spearman = 0.6323\n",
            "2022-04-23 04:55:12,396 : headlines : pearson = 0.7980, spearman = 0.8054\n",
            "2022-04-23 04:55:15,345 : OnWN : pearson = 0.8729, spearman = 0.8483\n",
            "2022-04-23 04:55:15,347 : ALL : Pearson = 0.8271,             Spearman = 0.8343\n",
            "2022-04-23 04:55:15,347 : ALL (weighted average) : Pearson = 0.8045,             Spearman = 0.7997\n",
            "2022-04-23 04:55:15,347 : ALL (average) : Pearson = 0.7662,             Spearman = 0.7620\n",
            "\n",
            "2022-04-23 04:55:15,349 : ***** Transfer task : STS14 *****\n",
            "\n",
            "\n",
            "2022-04-23 04:55:18,363 : deft-forum : pearson = 0.5806, spearman = 0.5797\n",
            "2022-04-23 04:55:21,884 : deft-news : pearson = 0.7995, spearman = 0.7569\n",
            "2022-04-23 04:55:26,722 : headlines : pearson = 0.7833, spearman = 0.7754\n",
            "2022-04-23 04:55:31,062 : images : pearson = 0.8396, spearman = 0.8203\n",
            "2022-04-23 04:55:35,511 : OnWN : pearson = 0.8886, spearman = 0.8636\n",
            "2022-04-23 04:55:41,575 : tweet-news : pearson = 0.7789, spearman = 0.7175\n",
            "2022-04-23 04:55:41,579 : ALL : Pearson = 0.7897,             Spearman = 0.7549\n",
            "2022-04-23 04:55:41,579 : ALL (weighted average) : Pearson = 0.7917,             Spearman = 0.7655\n",
            "2022-04-23 04:55:41,579 : ALL (average) : Pearson = 0.7784,             Spearman = 0.7522\n",
            "\n",
            "2022-04-23 04:55:41,582 : ***** Transfer task : STS15 *****\n",
            "\n",
            "\n",
            "2022-04-23 04:55:45,824 : answers-forums : pearson = 0.7252, spearman = 0.7287\n",
            "2022-04-23 04:55:50,190 : answers-students : pearson = 0.7692, spearman = 0.7759\n",
            "2022-04-23 04:55:54,690 : belief : pearson = 0.8040, spearman = 0.8081\n",
            "2022-04-23 04:56:00,044 : headlines : pearson = 0.8283, spearman = 0.8385\n",
            "2022-04-23 04:56:04,533 : images : pearson = 0.8817, spearman = 0.8983\n",
            "2022-04-23 04:56:04,535 : ALL : Pearson = 0.8184,             Spearman = 0.8281\n",
            "2022-04-23 04:56:04,535 : ALL (weighted average) : Pearson = 0.8109,             Spearman = 0.8203\n",
            "2022-04-23 04:56:04,536 : ALL (average) : Pearson = 0.8017,             Spearman = 0.8099\n",
            "\n",
            "2022-04-23 04:56:04,539 : ***** Transfer task : STS16 *****\n",
            "\n",
            "\n",
            "2022-04-23 04:56:06,466 : answer-answer : pearson = 0.7792, spearman = 0.7839\n",
            "2022-04-23 04:56:07,948 : headlines : pearson = 0.8059, spearman = 0.8345\n",
            "2022-04-23 04:56:09,690 : plagiarism : pearson = 0.8504, spearman = 0.8647\n",
            "2022-04-23 04:56:13,129 : postediting : pearson = 0.8571, spearman = 0.8775\n",
            "2022-04-23 04:56:14,503 : question-question : pearson = 0.7399, spearman = 0.7414\n",
            "2022-04-23 04:56:14,505 : ALL : Pearson = 0.8046,             Spearman = 0.8212\n",
            "2022-04-23 04:56:14,505 : ALL (weighted average) : Pearson = 0.8077,             Spearman = 0.8220\n",
            "2022-04-23 04:56:14,505 : ALL (average) : Pearson = 0.8065,             Spearman = 0.8204\n",
            "\n",
            "2022-04-23 04:56:14,507 : \n",
            "\n",
            "***** Transfer task : STSBenchmark*****\n",
            "\n",
            "\n",
            "2022-04-23 04:57:03,700 : train : pearson = 0.8118, spearman = 0.7974\n",
            "2022-04-23 04:57:17,725 : dev : pearson = 0.8399, spearman = 0.8442\n",
            "2022-04-23 04:57:29,655 : test : pearson = 0.8241, spearman = 0.8238\n",
            "2022-04-23 04:57:29,662 : ALL : Pearson = 0.8197,             Spearman = 0.8131\n",
            "2022-04-23 04:57:29,662 : ALL (weighted average) : Pearson = 0.8186,             Spearman = 0.8098\n",
            "2022-04-23 04:57:29,663 : ALL (average) : Pearson = 0.8253,             Spearman = 0.8218\n",
            "\n",
            "2022-04-23 04:57:29,668 : \n",
            "\n",
            "***** Transfer task : SICKRelatedness*****\n",
            "\n",
            "\n",
            "2022-04-23 04:57:59,373 : train : pearson = 0.7889, spearman = 0.7170\n",
            "2022-04-23 04:58:03,146 : dev : pearson = 0.7817, spearman = 0.7221\n",
            "2022-04-23 04:58:35,668 : test : pearson = 0.7828, spearman = 0.7119\n",
            "2022-04-23 04:58:35,674 : ALL : Pearson = 0.7856,             Spearman = 0.7147\n",
            "2022-04-23 04:58:35,675 : ALL (weighted average) : Pearson = 0.7855,             Spearman = 0.7147\n",
            "2022-04-23 04:58:35,675 : ALL (average) : Pearson = 0.7845,             Spearman = 0.7170\n",
            "\n",
            "------ test ------\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+-------+\n",
            "| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness |  Avg. |\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+-------+\n",
            "| 70.05 | 83.43 | 75.49 | 82.81 | 82.12 |    82.38     |      71.19      | 78.21 |\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+-------+\n",
            "+------+------+------+------+------+------+------+------+\n",
            "|  MR  |  CR  | SUBJ | MPQA | SST2 | TREC | MRPC | Avg. |\n",
            "+------+------+------+------+------+------+------+------+\n",
            "| 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |\n",
            "+------+------+------+------+------+------+------+------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluate Transfer Tasks for BERT/RoBERTa\n",
        "\n",
        "Running transfer task evaluation for BERT (`voidism/diffcse-bert-base-uncased-trans`) and RoBERTa (`voidism/diffcse-roberta-base-trans`).\n",
        "\n",
        "\n",
        "You should get the results on Table 2 in our paper: \n",
        "* DiffCSE-BERT: Avg. 86.86\n",
        "* DiffCSE-RoBERTa: Avg. 87.04"
      ],
      "metadata": {
        "id": "UGk7TWzSu5ZO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cd DiffCSE && python evaluation.py --model_name_or_path voidism/diffcse-bert-base-uncased-trans --pooler cls_before_pooler --task_set transfer --mode test"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "655USbZcu8H4",
        "outputId": "1fd00443-9362-48c5-c549-22ec8adca6c1"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-04-23 18:11:14,555 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:11:14,940 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-trans/resolve/main/config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 18:11:14,941 : Attempting to acquire lock 139939457998800 on /root/.cache/huggingface/transformers/62301d126d38cd7550d2b7e8ded1c38e1f326f7ed60229d2432763c5d11cd1ea.05453bc462024ebd07dcd59bd768abcf37dc137883087d74fba36bcd7e4a82ff.lock\n",
            "2022-04-23 18:11:14,941 : Lock 139939457998800 acquired on /root/.cache/huggingface/transformers/62301d126d38cd7550d2b7e8ded1c38e1f326f7ed60229d2432763c5d11cd1ea.05453bc462024ebd07dcd59bd768abcf37dc137883087d74fba36bcd7e4a82ff.lock\n",
            "2022-04-23 18:11:14,943 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:11:15,304 : https://huggingface.co:443 \"GET /voidism/diffcse-bert-base-uncased-trans/resolve/main/config.json HTTP/1.1\" 200 621\n",
            "Downloading: 100% 621/621 [00:00<00:00, 587kB/s]\n",
            "2022-04-23 18:11:15,306 : Attempting to release lock 139939457998800 on /root/.cache/huggingface/transformers/62301d126d38cd7550d2b7e8ded1c38e1f326f7ed60229d2432763c5d11cd1ea.05453bc462024ebd07dcd59bd768abcf37dc137883087d74fba36bcd7e4a82ff.lock\n",
            "2022-04-23 18:11:15,306 : Lock 139939457998800 released on /root/.cache/huggingface/transformers/62301d126d38cd7550d2b7e8ded1c38e1f326f7ed60229d2432763c5d11cd1ea.05453bc462024ebd07dcd59bd768abcf37dc137883087d74fba36bcd7e4a82ff.lock\n",
            "2022-04-23 18:11:15,308 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:11:15,670 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-trans/resolve/main/pytorch_model.bin HTTP/1.1\" 302 0\n",
            "2022-04-23 18:11:15,673 : Attempting to acquire lock 139939457998224 on /root/.cache/huggingface/transformers/c5bd812b54c660027aa10c3c57b9eb21312ee71dff497035818e22cdf0debbf9.6610679ef2155191bb23af0cd7db750c9934016fad82828ad279b9b616337dad.lock\n",
            "2022-04-23 18:11:15,673 : Lock 139939457998224 acquired on /root/.cache/huggingface/transformers/c5bd812b54c660027aa10c3c57b9eb21312ee71dff497035818e22cdf0debbf9.6610679ef2155191bb23af0cd7db750c9934016fad82828ad279b9b616337dad.lock\n",
            "2022-04-23 18:11:15,674 : Starting new HTTPS connection (1): cdn-lfs.huggingface.co:443\n",
            "2022-04-23 18:11:16,237 : https://cdn-lfs.huggingface.co:443 \"GET /repos/85/8a/858a6f6c567e215db01a7c6cff5aac39ca3b61bf35756200ccc5a8ce6950474d/8edb7ee1824ad1843a1f8edd0907b75c6bfb0d230052341c5347cb7e0705d880 HTTP/1.1\" 200 1245026652\n",
            "Downloading: 100% 1.25G/1.25G [00:40<00:00, 30.4MB/s]\n",
            "2022-04-23 18:11:57,218 : Attempting to release lock 139939457998224 on /root/.cache/huggingface/transformers/c5bd812b54c660027aa10c3c57b9eb21312ee71dff497035818e22cdf0debbf9.6610679ef2155191bb23af0cd7db750c9934016fad82828ad279b9b616337dad.lock\n",
            "2022-04-23 18:11:57,218 : Lock 139939457998224 released on /root/.cache/huggingface/transformers/c5bd812b54c660027aa10c3c57b9eb21312ee71dff497035818e22cdf0debbf9.6610679ef2155191bb23af0cd7db750c9934016fad82828ad279b9b616337dad.lock\n",
            "Some weights of BertModel were not initialized from the model checkpoint at voidism/diffcse-bert-base-uncased-trans and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
            "2022-04-23 18:11:59,989 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:12:00,358 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-trans/resolve/main/config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 18:12:00,362 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:12:00,723 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-trans/resolve/main/vocab.txt HTTP/1.1\" 200 0\n",
            "2022-04-23 18:12:00,725 : Attempting to acquire lock 139939457168528 on /root/.cache/huggingface/transformers/3a5e604531aaa5a2dc2adcce8d361f591d421ce9b7ea2a989d5e1c60c46bfb4b.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock\n",
            "2022-04-23 18:12:00,725 : Lock 139939457168528 acquired on /root/.cache/huggingface/transformers/3a5e604531aaa5a2dc2adcce8d361f591d421ce9b7ea2a989d5e1c60c46bfb4b.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock\n",
            "2022-04-23 18:12:00,726 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:12:01,173 : https://huggingface.co:443 \"GET /voidism/diffcse-bert-base-uncased-trans/resolve/main/vocab.txt HTTP/1.1\" 200 231508\n",
            "Downloading: 100% 232k/232k [00:00<00:00, 953kB/s] \n",
            "2022-04-23 18:12:01,426 : Attempting to release lock 139939457168528 on /root/.cache/huggingface/transformers/3a5e604531aaa5a2dc2adcce8d361f591d421ce9b7ea2a989d5e1c60c46bfb4b.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock\n",
            "2022-04-23 18:12:01,426 : Lock 139939457168528 released on /root/.cache/huggingface/transformers/3a5e604531aaa5a2dc2adcce8d361f591d421ce9b7ea2a989d5e1c60c46bfb4b.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock\n",
            "2022-04-23 18:12:01,427 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:12:01,789 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-trans/resolve/main/tokenizer.json HTTP/1.1\" 404 0\n",
            "2022-04-23 18:12:01,792 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:12:02,152 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-trans/resolve/main/added_tokens.json HTTP/1.1\" 404 0\n",
            "2022-04-23 18:12:02,154 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:12:02,514 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-trans/resolve/main/special_tokens_map.json HTTP/1.1\" 404 0\n",
            "2022-04-23 18:12:02,516 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 18:12:02,877 : https://huggingface.co:443 \"HEAD /voidism/diffcse-bert-base-uncased-trans/resolve/main/tokenizer_config.json HTTP/1.1\" 404 0\n",
            "2022-04-23 18:12:05,319 : ***** Transfer task : MR *****\n",
            "\n",
            "\n",
            "2022-04-23 18:12:05,461 : Generating sentence embeddings\n",
            "2022-04-23 18:12:19,424 : Generated sentence embeddings\n",
            "2022-04-23 18:12:19,425 : Training pytorch-MLP-nhid0-adam-bs64 with (inner) 10-fold cross-validation\n",
            "2022-04-23 18:15:21,762 : Best param found at split 1: l2reg = 0.001                 with score 83.3\n",
            "2022-04-23 18:18:22,913 : Best param found at split 2: l2reg = 0.001                 with score 83.66\n",
            "2022-04-23 18:21:23,882 : Best param found at split 3: l2reg = 1e-05                 with score 83.25\n",
            "2022-04-23 18:24:40,353 : Best param found at split 4: l2reg = 0.0001                 with score 83.24\n",
            "2022-04-23 18:27:33,851 : Best param found at split 5: l2reg = 0.001                 with score 83.42\n",
            "2022-04-23 18:30:41,617 : Best param found at split 6: l2reg = 0.001                 with score 83.31\n",
            "2022-04-23 18:33:45,368 : Best param found at split 7: l2reg = 0.001                 with score 83.24\n",
            "2022-04-23 18:36:44,622 : Best param found at split 8: l2reg = 0.0001                 with score 83.18\n",
            "2022-04-23 18:39:35,571 : Best param found at split 9: l2reg = 0.001                 with score 83.16\n",
            "2022-04-23 18:42:47,736 : Best param found at split 10: l2reg = 0.0001                 with score 83.09\n",
            "2022-04-23 18:42:52,107 : Dev acc : 83.28 Test acc : 82.69\n",
            "\n",
            "2022-04-23 18:42:52,113 : ***** Transfer task : CR *****\n",
            "\n",
            "\n",
            "2022-04-23 18:42:52,123 : Generating sentence embeddings\n",
            "2022-04-23 18:42:55,894 : Generated sentence embeddings\n",
            "2022-04-23 18:42:55,894 : Training pytorch-MLP-nhid0-adam-bs64 with (inner) 10-fold cross-validation\n",
            "2022-04-23 18:43:56,548 : Best param found at split 1: l2reg = 1e-05                 with score 88.49\n",
            "2022-04-23 18:45:00,712 : Best param found at split 2: l2reg = 1e-05                 with score 88.31\n",
            "2022-04-23 18:46:10,347 : Best param found at split 3: l2reg = 1e-05                 with score 88.31\n",
            "2022-04-23 18:47:15,796 : Best param found at split 4: l2reg = 0.001                 with score 88.55\n",
            "2022-04-23 18:48:16,995 : Best param found at split 5: l2reg = 1e-05                 with score 87.96\n",
            "2022-04-23 18:49:21,486 : Best param found at split 6: l2reg = 1e-05                 with score 88.08\n",
            "2022-04-23 18:50:26,798 : Best param found at split 7: l2reg = 0.0001                 with score 87.99\n",
            "2022-04-23 18:51:34,064 : Best param found at split 8: l2reg = 0.0001                 with score 88.08\n",
            "2022-04-23 18:52:37,356 : Best param found at split 9: l2reg = 1e-05                 with score 87.87\n",
            "2022-04-23 18:53:40,026 : Best param found at split 10: l2reg = 1e-05                 with score 88.2\n",
            "2022-04-23 18:53:41,415 : Dev acc : 88.18 Test acc : 87.23\n",
            "\n",
            "2022-04-23 18:53:41,417 : ***** Transfer task : MPQA *****\n",
            "\n",
            "\n",
            "2022-04-23 18:53:41,430 : Generating sentence embeddings\n",
            "2022-04-23 18:53:44,883 : Generated sentence embeddings\n",
            "2022-04-23 18:53:44,884 : Training pytorch-MLP-nhid0-adam-bs64 with (inner) 10-fold cross-validation\n",
            "2022-04-23 18:56:48,644 : Best param found at split 1: l2reg = 0.0001                 with score 89.97\n",
            "2022-04-23 19:00:03,840 : Best param found at split 2: l2reg = 1e-05                 with score 89.96\n",
            "2022-04-23 19:03:20,462 : Best param found at split 3: l2reg = 1e-05                 with score 89.56\n",
            "2022-04-23 19:06:28,921 : Best param found at split 4: l2reg = 0.0001                 with score 89.64\n",
            "2022-04-23 19:09:29,289 : Best param found at split 5: l2reg = 1e-05                 with score 89.79\n",
            "2022-04-23 19:12:31,517 : Best param found at split 6: l2reg = 0.0001                 with score 89.76\n",
            "2022-04-23 19:15:43,522 : Best param found at split 7: l2reg = 1e-05                 with score 89.91\n",
            "2022-04-23 19:19:01,349 : Best param found at split 8: l2reg = 1e-05                 with score 89.96\n",
            "2022-04-23 19:22:09,203 : Best param found at split 9: l2reg = 1e-05                 with score 89.77\n",
            "2022-04-23 19:25:15,189 : Best param found at split 10: l2reg = 1e-05                 with score 89.72\n",
            "2022-04-23 19:25:18,592 : Dev acc : 89.8 Test acc : 89.28\n",
            "\n",
            "2022-04-23 19:25:18,594 : ***** Transfer task : SUBJ *****\n",
            "\n",
            "\n",
            "2022-04-23 19:25:18,627 : Generating sentence embeddings\n",
            "2022-04-23 19:25:32,386 : Generated sentence embeddings\n",
            "2022-04-23 19:25:32,387 : Training pytorch-MLP-nhid0-adam-bs64 with (inner) 10-fold cross-validation\n",
            "2022-04-23 19:28:37,652 : Best param found at split 1: l2reg = 1e-05                 with score 95.66\n",
            "2022-04-23 19:31:37,818 : Best param found at split 2: l2reg = 0.001                 with score 95.54\n",
            "2022-04-23 19:34:45,070 : Best param found at split 3: l2reg = 0.0001                 with score 95.73\n",
            "2022-04-23 19:37:43,790 : Best param found at split 4: l2reg = 0.001                 with score 95.59\n",
            "2022-04-23 19:40:50,415 : Best param found at split 5: l2reg = 0.0001                 with score 95.7\n",
            "2022-04-23 19:43:46,837 : Best param found at split 6: l2reg = 1e-05                 with score 95.53\n",
            "2022-04-23 19:46:55,635 : Best param found at split 7: l2reg = 1e-05                 with score 95.88\n",
            "2022-04-23 19:49:54,839 : Best param found at split 8: l2reg = 1e-05                 with score 95.73\n",
            "2022-04-23 19:53:04,826 : Best param found at split 9: l2reg = 0.0001                 with score 95.72\n",
            "2022-04-23 19:56:13,055 : Best param found at split 10: l2reg = 0.0001                 with score 95.66\n",
            "2022-04-23 19:56:18,113 : Dev acc : 95.67 Test acc : 95.23\n",
            "\n",
            "2022-04-23 19:56:18,119 : ***** Transfer task : SST Binary classification *****\n",
            "\n",
            "\n",
            "2022-04-23 19:56:18,312 : Computing embedding for train\n",
            "2022-04-23 19:57:03,898 : Computed train embeddings\n",
            "2022-04-23 19:57:03,898 : Computing embedding for dev\n",
            "2022-04-23 19:57:04,895 : Computed dev embeddings\n",
            "2022-04-23 19:57:04,895 : Computing embedding for test\n",
            "2022-04-23 19:57:06,961 : Computed test embeddings\n",
            "2022-04-23 19:57:06,961 : Training pytorch-MLP-nhid0-adam-bs64 with standard validation..\n",
            "2022-04-23 19:59:02,918 : [('reg:1e-05', 88.3), ('reg:0.0001', 88.53), ('reg:0.001', 88.3), ('reg:0.01', 86.93)]\n",
            "2022-04-23 19:59:02,919 : Validation : best param found is reg = 0.0001 with score             88.53\n",
            "2022-04-23 19:59:02,919 : Evaluating...\n",
            "2022-04-23 19:59:36,301 : \n",
            "Dev acc : 88.53 Test acc : 86.6 for             SST Binary classification\n",
            "\n",
            "2022-04-23 19:59:36,339 : ***** Transfer task : TREC *****\n",
            "\n",
            "\n",
            "2022-04-23 19:59:39,969 : Computed train embeddings\n",
            "2022-04-23 19:59:40,241 : Computed test embeddings\n",
            "2022-04-23 19:59:40,242 : Training pytorch-MLP-nhid0-adam-bs64 with 10-fold cross-validation\n",
            "2022-04-23 20:01:58,821 : [('reg:1e-05', 86.65), ('reg:0.0001', 86.59), ('reg:0.001', 86.43), ('reg:0.01', 83.58)]\n",
            "2022-04-23 20:01:58,821 : Cross-validation : best param found is reg = 1e-05             with score 86.65\n",
            "2022-04-23 20:01:58,821 : Evaluating...\n",
            "2022-04-23 20:02:03,050 : \n",
            "Dev acc : 86.65 Test acc : 90.4             for TREC\n",
            "\n",
            "2022-04-23 20:02:03,052 : ***** Transfer task : MRPC *****\n",
            "\n",
            "\n",
            "2022-04-23 20:02:03,089 : Computing embedding for train\n",
            "2022-04-23 20:02:13,132 : Computed train embeddings\n",
            "2022-04-23 20:02:13,132 : Computing embedding for test\n",
            "2022-04-23 20:02:17,480 : Computed test embeddings\n",
            "2022-04-23 20:02:17,506 : Training pytorch-MLP-nhid0-adam-bs64 with 10-fold cross-validation\n",
            "2022-04-23 20:03:35,465 : [('reg:1e-05', 76.89), ('reg:0.0001', 76.84), ('reg:0.001', 76.72), ('reg:0.01', 75.86)]\n",
            "2022-04-23 20:03:35,465 : Cross-validation : best param found is reg = 1e-05             with score 76.89\n",
            "2022-04-23 20:03:35,465 : Evaluating...\n",
            "2022-04-23 20:03:37,395 : Dev acc : 76.89 Test acc 76.58; Test F1 83.71 for MRPC.\n",
            "\n",
            "------ test ------\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+------+\n",
            "| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+------+\n",
            "|  0.00 |  0.00 |  0.00 |  0.00 |  0.00 |     0.00     |       0.00      | 0.00 |\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+------+\n",
            "+-------+-------+-------+-------+-------+-------+-------+-------+\n",
            "|   MR  |   CR  |  SUBJ |  MPQA |  SST2 |  TREC |  MRPC |  Avg. |\n",
            "+-------+-------+-------+-------+-------+-------+-------+-------+\n",
            "| 82.69 | 87.23 | 95.23 | 89.28 | 86.60 | 90.40 | 76.58 | 86.86 |\n",
            "+-------+-------+-------+-------+-------+-------+-------+-------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!cd DiffCSE && python evaluation.py --model_name_or_path voidism/diffcse-roberta-base-trans --pooler cls_before_pooler --task_set transfer --mode test"
      ],
      "metadata": {
        "id": "QHp0FI_Kv2C5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "84d5f1e9-f6a8-41df-b1a4-1d979fd6e063"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-04-23 20:03:39,372 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:03:39,747 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 20:03:39,749 : Attempting to acquire lock 140368544131920 on /root/.cache/huggingface/transformers/cbc0216c5696b28401b8c2a9e27ae0929aaad2cae461ff46bd447b685df1cc35.eefe78f596ddb431de5f2137a090dd98ca6b8f3868c14a81d5a0892a09fbce6d.lock\n",
            "2022-04-23 20:03:39,749 : Lock 140368544131920 acquired on /root/.cache/huggingface/transformers/cbc0216c5696b28401b8c2a9e27ae0929aaad2cae461ff46bd447b685df1cc35.eefe78f596ddb431de5f2137a090dd98ca6b8f3868c14a81d5a0892a09fbce6d.lock\n",
            "2022-04-23 20:03:39,751 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:03:40,111 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-trans/resolve/main/config.json HTTP/1.1\" 200 664\n",
            "Downloading: 100% 664/664 [00:00<00:00, 556kB/s]\n",
            "2022-04-23 20:03:40,114 : Attempting to release lock 140368544131920 on /root/.cache/huggingface/transformers/cbc0216c5696b28401b8c2a9e27ae0929aaad2cae461ff46bd447b685df1cc35.eefe78f596ddb431de5f2137a090dd98ca6b8f3868c14a81d5a0892a09fbce6d.lock\n",
            "2022-04-23 20:03:40,114 : Lock 140368544131920 released on /root/.cache/huggingface/transformers/cbc0216c5696b28401b8c2a9e27ae0929aaad2cae461ff46bd447b685df1cc35.eefe78f596ddb431de5f2137a090dd98ca6b8f3868c14a81d5a0892a09fbce6d.lock\n",
            "2022-04-23 20:03:40,116 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:03:40,477 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/pytorch_model.bin HTTP/1.1\" 302 0\n",
            "2022-04-23 20:03:40,480 : Attempting to acquire lock 140368544131920 on /root/.cache/huggingface/transformers/c6f83bb1419c79e820fb92687ecdf1a1efbc7cb4c9c214a8a7a8baf086a4a9ed.6363d37ff10232415735d4052ec343237d1fd1a70a52747bc6b4b66be1363141.lock\n",
            "2022-04-23 20:03:40,480 : Lock 140368544131920 acquired on /root/.cache/huggingface/transformers/c6f83bb1419c79e820fb92687ecdf1a1efbc7cb4c9c214a8a7a8baf086a4a9ed.6363d37ff10232415735d4052ec343237d1fd1a70a52747bc6b4b66be1363141.lock\n",
            "2022-04-23 20:03:40,481 : Starting new HTTPS connection (1): cdn-lfs.huggingface.co:443\n",
            "2022-04-23 20:03:41,044 : https://cdn-lfs.huggingface.co:443 \"GET /repos/e1/55/e15544af37c7f1f4bce4b3c6882d6f08b9ad7e78e9d59cf79ec13f4cef0a8e63/887994f8b60a5e72c2b3063b05472612ef439c3862d0f887262296a8775e206c HTTP/1.1\" 200 1487809454\n",
            "Downloading: 100% 1.49G/1.49G [01:12<00:00, 20.6MB/s]\n",
            "2022-04-23 20:04:53,363 : Attempting to release lock 140368544131920 on /root/.cache/huggingface/transformers/c6f83bb1419c79e820fb92687ecdf1a1efbc7cb4c9c214a8a7a8baf086a4a9ed.6363d37ff10232415735d4052ec343237d1fd1a70a52747bc6b4b66be1363141.lock\n",
            "2022-04-23 20:04:53,363 : Lock 140368544131920 released on /root/.cache/huggingface/transformers/c6f83bb1419c79e820fb92687ecdf1a1efbc7cb4c9c214a8a7a8baf086a4a9ed.6363d37ff10232415735d4052ec343237d1fd1a70a52747bc6b4b66be1363141.lock\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at voidism/diffcse-roberta-base-trans and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
            "2022-04-23 20:04:56,675 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:04:57,047 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 20:04:57,050 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:04:57,422 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/vocab.json HTTP/1.1\" 200 0\n",
            "2022-04-23 20:04:57,423 : Attempting to acquire lock 140368546715024 on /root/.cache/huggingface/transformers/384071b90fbb2536b6a38d502cf4cbbf1c926006562e544920d5a530a3113e52.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7.lock\n",
            "2022-04-23 20:04:57,424 : Lock 140368546715024 acquired on /root/.cache/huggingface/transformers/384071b90fbb2536b6a38d502cf4cbbf1c926006562e544920d5a530a3113e52.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7.lock\n",
            "2022-04-23 20:04:57,425 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:04:57,799 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-trans/resolve/main/vocab.json HTTP/1.1\" 200 798293\n",
            "Downloading: 100% 798k/798k [00:00<00:00, 1.88MB/s]\n",
            "2022-04-23 20:04:58,225 : Attempting to release lock 140368546715024 on /root/.cache/huggingface/transformers/384071b90fbb2536b6a38d502cf4cbbf1c926006562e544920d5a530a3113e52.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7.lock\n",
            "2022-04-23 20:04:58,225 : Lock 140368546715024 released on /root/.cache/huggingface/transformers/384071b90fbb2536b6a38d502cf4cbbf1c926006562e544920d5a530a3113e52.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7.lock\n",
            "2022-04-23 20:04:58,226 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:04:58,587 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/merges.txt HTTP/1.1\" 200 0\n",
            "2022-04-23 20:04:58,589 : Attempting to acquire lock 140368520718288 on /root/.cache/huggingface/transformers/65a5f19ebcbe9a9a5d8795f7d5948e09533d688d952c64c56e19b7ddc1c2bc0b.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435.lock\n",
            "2022-04-23 20:04:58,589 : Lock 140368520718288 acquired on /root/.cache/huggingface/transformers/65a5f19ebcbe9a9a5d8795f7d5948e09533d688d952c64c56e19b7ddc1c2bc0b.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435.lock\n",
            "2022-04-23 20:04:58,590 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:04:58,972 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-trans/resolve/main/merges.txt HTTP/1.1\" 200 456356\n",
            "Downloading: 100% 456k/456k [00:00<00:00, 1.08MB/s]\n",
            "2022-04-23 20:04:59,397 : Attempting to release lock 140368520718288 on /root/.cache/huggingface/transformers/65a5f19ebcbe9a9a5d8795f7d5948e09533d688d952c64c56e19b7ddc1c2bc0b.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435.lock\n",
            "2022-04-23 20:04:59,397 : Lock 140368520718288 released on /root/.cache/huggingface/transformers/65a5f19ebcbe9a9a5d8795f7d5948e09533d688d952c64c56e19b7ddc1c2bc0b.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435.lock\n",
            "2022-04-23 20:04:59,399 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:04:59,757 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/tokenizer.json HTTP/1.1\" 404 0\n",
            "2022-04-23 20:04:59,760 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:05:00,122 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/added_tokens.json HTTP/1.1\" 404 0\n",
            "2022-04-23 20:05:00,125 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:05:00,489 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/special_tokens_map.json HTTP/1.1\" 200 0\n",
            "2022-04-23 20:05:00,491 : Attempting to acquire lock 140368520743184 on /root/.cache/huggingface/transformers/991d995c20df9b34cfdc8a407c8c0028ed1b3b7270fcbe60ff607f0554f67aba.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342.lock\n",
            "2022-04-23 20:05:00,491 : Lock 140368520743184 acquired on /root/.cache/huggingface/transformers/991d995c20df9b34cfdc8a407c8c0028ed1b3b7270fcbe60ff607f0554f67aba.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342.lock\n",
            "2022-04-23 20:05:00,492 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:05:00,852 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-trans/resolve/main/special_tokens_map.json HTTP/1.1\" 200 239\n",
            "Downloading: 100% 239/239 [00:00<00:00, 194kB/s]\n",
            "2022-04-23 20:05:00,854 : Attempting to release lock 140368520743184 on /root/.cache/huggingface/transformers/991d995c20df9b34cfdc8a407c8c0028ed1b3b7270fcbe60ff607f0554f67aba.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342.lock\n",
            "2022-04-23 20:05:00,854 : Lock 140368520743184 released on /root/.cache/huggingface/transformers/991d995c20df9b34cfdc8a407c8c0028ed1b3b7270fcbe60ff607f0554f67aba.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342.lock\n",
            "2022-04-23 20:05:00,856 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:05:01,215 : https://huggingface.co:443 \"HEAD /voidism/diffcse-roberta-base-trans/resolve/main/tokenizer_config.json HTTP/1.1\" 200 0\n",
            "2022-04-23 20:05:01,216 : Attempting to acquire lock 140368520758480 on /root/.cache/huggingface/transformers/01e905495f7f5b3a702d008ee0d5e225b6d8dbfd93b4c32e48b25f1a4ebe38cc.3fd6ad28580f134ac5fac0b861423fd64502e7b6abeb15831037d02b89280db7.lock\n",
            "2022-04-23 20:05:01,217 : Lock 140368520758480 acquired on /root/.cache/huggingface/transformers/01e905495f7f5b3a702d008ee0d5e225b6d8dbfd93b4c32e48b25f1a4ebe38cc.3fd6ad28580f134ac5fac0b861423fd64502e7b6abeb15831037d02b89280db7.lock\n",
            "2022-04-23 20:05:01,218 : Starting new HTTPS connection (1): huggingface.co:443\n",
            "2022-04-23 20:05:01,578 : https://huggingface.co:443 \"GET /voidism/diffcse-roberta-base-trans/resolve/main/tokenizer_config.json HTTP/1.1\" 200 265\n",
            "Downloading: 100% 265/265 [00:00<00:00, 219kB/s]\n",
            "2022-04-23 20:05:01,580 : Attempting to release lock 140368520758480 on /root/.cache/huggingface/transformers/01e905495f7f5b3a702d008ee0d5e225b6d8dbfd93b4c32e48b25f1a4ebe38cc.3fd6ad28580f134ac5fac0b861423fd64502e7b6abeb15831037d02b89280db7.lock\n",
            "2022-04-23 20:05:01,580 : Lock 140368520758480 released on /root/.cache/huggingface/transformers/01e905495f7f5b3a702d008ee0d5e225b6d8dbfd93b4c32e48b25f1a4ebe38cc.3fd6ad28580f134ac5fac0b861423fd64502e7b6abeb15831037d02b89280db7.lock\n",
            "2022-04-23 20:05:04,845 : ***** Transfer task : MR *****\n",
            "\n",
            "\n",
            "2022-04-23 20:05:05,007 : Generating sentence embeddings\n",
            "2022-04-23 20:05:19,114 : Generated sentence embeddings\n",
            "2022-04-23 20:05:19,115 : Training pytorch-MLP-nhid0-adam-bs64 with (inner) 10-fold cross-validation\n",
            "2022-04-23 20:08:22,925 : Best param found at split 1: l2reg = 1e-05                 with score 83.43\n",
            "2022-04-23 20:11:31,648 : Best param found at split 2: l2reg = 0.0001                 with score 83.84\n",
            "2022-04-23 20:14:35,496 : Best param found at split 3: l2reg = 0.01                 with score 83.64\n",
            "2022-04-23 20:17:42,507 : Best param found at split 4: l2reg = 0.0001                 with score 83.71\n",
            "2022-04-23 20:20:29,861 : Best param found at split 5: l2reg = 1e-05                 with score 83.57\n",
            "2022-04-23 20:23:41,275 : Best param found at split 6: l2reg = 0.001                 with score 83.46\n",
            "2022-04-23 20:26:38,496 : Best param found at split 7: l2reg = 0.0001                 with score 83.47\n",
            "2022-04-23 20:29:46,602 : Best param found at split 8: l2reg = 0.0001                 with score 83.4\n",
            "2022-04-23 20:32:34,627 : Best param found at split 9: l2reg = 1e-05                 with score 83.36\n",
            "2022-04-23 20:35:24,094 : Best param found at split 10: l2reg = 0.01                 with score 83.42\n",
            "2022-04-23 20:35:28,585 : Dev acc : 83.53 Test acc : 82.82\n",
            "\n",
            "2022-04-23 20:35:28,590 : ***** Transfer task : CR *****\n",
            "\n",
            "\n",
            "2022-04-23 20:35:28,599 : Generating sentence embeddings\n",
            "2022-04-23 20:35:32,323 : Generated sentence embeddings\n",
            "2022-04-23 20:35:32,323 : Training pytorch-MLP-nhid0-adam-bs64 with (inner) 10-fold cross-validation\n",
            "2022-04-23 20:36:33,348 : Best param found at split 1: l2reg = 1e-05                 with score 89.93\n",
            "2022-04-23 20:37:35,164 : Best param found at split 2: l2reg = 1e-05                 with score 89.31\n",
            "2022-04-23 20:38:41,071 : Best param found at split 3: l2reg = 1e-05                 with score 89.73\n",
            "2022-04-23 20:39:43,013 : Best param found at split 4: l2reg = 1e-05                 with score 89.93\n",
            "2022-04-23 20:40:47,599 : Best param found at split 5: l2reg = 0.001                 with score 89.22\n",
            "2022-04-23 20:41:50,337 : Best param found at split 6: l2reg = 0.001                 with score 89.23\n",
            "2022-04-23 20:42:56,227 : Best param found at split 7: l2reg = 1e-05                 with score 89.49\n",
            "2022-04-23 20:43:58,870 : Best param found at split 8: l2reg = 0.01                 with score 89.58\n",
            "2022-04-23 20:45:05,388 : Best param found at split 9: l2reg = 1e-05                 with score 89.32\n",
            "2022-04-23 20:46:08,914 : Best param found at split 10: l2reg = 1e-05                 with score 89.79\n",
            "2022-04-23 20:46:10,336 : Dev acc : 89.55 Test acc : 88.61\n",
            "\n",
            "2022-04-23 20:46:10,338 : ***** Transfer task : MPQA *****\n",
            "\n",
            "\n",
            "2022-04-23 20:46:10,403 : Generating sentence embeddings\n",
            "2022-04-23 20:46:13,723 : Generated sentence embeddings\n",
            "2022-04-23 20:46:13,724 : Training pytorch-MLP-nhid0-adam-bs64 with (inner) 10-fold cross-validation\n",
            "2022-04-23 20:49:01,868 : Best param found at split 1: l2reg = 0.01                 with score 88.34\n",
            "2022-04-23 20:51:47,669 : Best param found at split 2: l2reg = 0.0001                 with score 88.14\n",
            "2022-04-23 20:54:44,815 : Best param found at split 3: l2reg = 0.001                 with score 88.24\n",
            "2022-04-23 20:57:53,140 : Best param found at split 4: l2reg = 0.01                 with score 88.34\n",
            "2022-04-23 21:00:58,387 : Best param found at split 5: l2reg = 0.01                 with score 88.27\n",
            "2022-04-23 21:04:00,358 : Best param found at split 6: l2reg = 0.0001                 with score 88.17\n",
            "2022-04-23 21:07:10,625 : Best param found at split 7: l2reg = 0.001                 with score 88.52\n",
            "2022-04-23 21:10:06,642 : Best param found at split 8: l2reg = 0.01                 with score 88.17\n",
            "2022-04-23 21:13:19,762 : Best param found at split 9: l2reg = 0.01                 with score 88.15\n",
            "2022-04-23 21:16:17,147 : Best param found at split 10: l2reg = 0.001                 with score 88.05\n",
            "2022-04-23 21:16:23,037 : Dev acc : 88.24 Test acc : 87.71\n",
            "\n",
            "2022-04-23 21:16:23,039 : ***** Transfer task : SUBJ *****\n",
            "\n",
            "\n",
            "2022-04-23 21:16:23,070 : Generating sentence embeddings\n",
            "2022-04-23 21:16:36,940 : Generated sentence embeddings\n",
            "2022-04-23 21:16:36,940 : Training pytorch-MLP-nhid0-adam-bs64 with (inner) 10-fold cross-validation\n",
            "2022-04-23 21:19:24,665 : Best param found at split 1: l2reg = 0.0001                 with score 94.39\n",
            "2022-04-23 21:22:25,714 : Best param found at split 2: l2reg = 1e-05                 with score 94.43\n",
            "2022-04-23 21:25:32,671 : Best param found at split 3: l2reg = 1e-05                 with score 94.89\n",
            "2022-04-23 21:28:43,109 : Best param found at split 4: l2reg = 1e-05                 with score 94.64\n",
            "2022-04-23 21:31:38,500 : Best param found at split 5: l2reg = 1e-05                 with score 94.59\n",
            "2022-04-23 21:34:48,685 : Best param found at split 6: l2reg = 1e-05                 with score 94.77\n",
            "2022-04-23 21:38:09,866 : Best param found at split 7: l2reg = 0.001                 with score 94.82\n",
            "2022-04-23 21:41:08,867 : Best param found at split 8: l2reg = 0.0001                 with score 94.51\n",
            "2022-04-23 21:44:28,579 : Best param found at split 9: l2reg = 1e-05                 with score 94.5\n",
            "2022-04-23 21:47:27,916 : Best param found at split 10: l2reg = 0.0001                 with score 94.44\n",
            "2022-04-23 21:47:32,095 : Dev acc : 94.6 Test acc : 94.32\n",
            "\n",
            "2022-04-23 21:47:32,100 : ***** Transfer task : SST Binary classification *****\n",
            "\n",
            "\n",
            "2022-04-23 21:47:32,284 : Computing embedding for train\n",
            "2022-04-23 21:48:17,867 : Computed train embeddings\n",
            "2022-04-23 21:48:17,868 : Computing embedding for dev\n",
            "2022-04-23 21:48:18,846 : Computed dev embeddings\n",
            "2022-04-23 21:48:18,846 : Computing embedding for test\n",
            "2022-04-23 21:48:20,861 : Computed test embeddings\n",
            "2022-04-23 21:48:20,861 : Training pytorch-MLP-nhid0-adam-bs64 with standard validation..\n",
            "2022-04-23 21:50:32,739 : [('reg:1e-05', 87.39), ('reg:0.0001', 87.39), ('reg:0.001', 87.39), ('reg:0.01', 87.27)]\n",
            "2022-04-23 21:50:32,739 : Validation : best param found is reg = 1e-05 with score             87.39\n",
            "2022-04-23 21:50:32,739 : Evaluating...\n",
            "2022-04-23 21:51:05,462 : \n",
            "Dev acc : 87.39 Test acc : 88.63 for             SST Binary classification\n",
            "\n",
            "2022-04-23 21:51:05,500 : ***** Transfer task : TREC *****\n",
            "\n",
            "\n",
            "2022-04-23 21:51:09,140 : Computed train embeddings\n",
            "2022-04-23 21:51:09,407 : Computed test embeddings\n",
            "2022-04-23 21:51:09,408 : Training pytorch-MLP-nhid0-adam-bs64 with 10-fold cross-validation\n",
            "2022-04-23 21:53:24,325 : [('reg:1e-05', 81.88), ('reg:0.0001', 81.91), ('reg:0.001', 81.84), ('reg:0.01', 80.81)]\n",
            "2022-04-23 21:53:24,325 : Cross-validation : best param found is reg = 0.0001             with score 81.91\n",
            "2022-04-23 21:53:24,326 : Evaluating...\n",
            "2022-04-23 21:53:28,033 : \n",
            "Dev acc : 81.91 Test acc : 90.4             for TREC\n",
            "\n",
            "2022-04-23 21:53:28,035 : ***** Transfer task : MRPC *****\n",
            "\n",
            "\n",
            "2022-04-23 21:53:28,064 : Computing embedding for train\n",
            "2022-04-23 21:53:38,047 : Computed train embeddings\n",
            "2022-04-23 21:53:38,047 : Computing embedding for test\n",
            "2022-04-23 21:53:42,419 : Computed test embeddings\n",
            "2022-04-23 21:53:42,444 : Training pytorch-MLP-nhid0-adam-bs64 with 10-fold cross-validation\n",
            "2022-04-23 21:54:55,170 : [('reg:1e-05', 77.5), ('reg:0.0001', 77.5), ('reg:0.001', 77.55), ('reg:0.01', 77.7)]\n",
            "2022-04-23 21:54:55,170 : Cross-validation : best param found is reg = 0.01             with score 77.7\n",
            "2022-04-23 21:54:55,170 : Evaluating...\n",
            "2022-04-23 21:54:56,963 : Dev acc : 77.7 Test acc 76.81; Test F1 83.39 for MRPC.\n",
            "\n",
            "------ test ------\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+------+\n",
            "| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+------+\n",
            "|  0.00 |  0.00 |  0.00 |  0.00 |  0.00 |     0.00     |       0.00      | 0.00 |\n",
            "+-------+-------+-------+-------+-------+--------------+-----------------+------+\n",
            "+-------+-------+-------+-------+-------+-------+-------+-------+\n",
            "|   MR  |   CR  |  SUBJ |  MPQA |  SST2 |  TREC |  MRPC |  Avg. |\n",
            "+-------+-------+-------+-------+-------+-------+-------+-------+\n",
            "| 82.82 | 88.61 | 94.32 | 87.71 | 88.63 | 90.40 | 76.81 | 87.04 |\n",
            "+-------+-------+-------+-------+-------+-------+-------+-------+\n"
          ]
        }
      ]
    }
  ]
}