L2_getting_started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ac959c4b-d7a9-4949-9c39-2ab46fbf82cd",
   "metadata": {},
   "source": [
    "# Lesson 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6df1e2b-2079-4dee-84e9-77e13dd31e05",
   "metadata": {},
   "source": [
    "### Getting started with Llama 2\n",
    "\n",
    "The code to call the Llama 2 models through the Together.ai hosted API service has been wrapped into a helper function called `llama`. You can take a look at this code if you like by opening the utils.py file using the File -> Open menu item above this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "10075542-6019-40b6-b3c1-532383e4a6dd",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "# import llama helper function\n",
    "from utils import llama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "03a549fd-dd50-4ae1-a5d8-ce00fdc707a3",
   "metadata": {
    "height": 58
   },
   "outputs": [],
   "source": [
    "# define the prompt\n",
    "prompt = \"Help me write a birthday card for my dear friend Andrew.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9f201c8e",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function utils.llama(prompt, add_inst=True, model='togethercomputer/llama-2-7b-chat', temperature=0.0, max_tokens=1024, verbose=False, url='http://jupyter-api-proxy.internal.dlai/rev-proxy/meta_together/inference', headers={'Authorization': 'Bearer eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhcHAiLCJzdWIiOiI2NjAxODYiLCJhdWQiOiJXRUIiLCJpYXQiOjE3MTAxNTI1NjEsImV4cCI6MTcxMjc0NDU2MX0.TCQBQTahSz4D0WUpXlVxDnPUasWgl85EQFfyiQ7LLWA', 'Content-Type': 'application/json'}, base=2, max_tries=3)>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llama"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16d46c4b-2926-44a9-9e25-befb6bd6648c",
   "metadata": {},
   "source": [
    "**Note:** LLMs can have different responses for the same prompt, which is why throughout the course, the responses you get might be slightly different than the ones in the lecture videos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b89f4c48-363d-4ddb-8f20-215bbd47004a",
   "metadata": {
    "height": 75
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Of course, I'd be happy to help you write a birthday card for your dear friend Andrew! Here are a few suggestions:\n",
      "\n",
      "1. Personalized Message: Start by writing a personalized message that speaks to your friendship with Andrew. You could mention a favorite memory or inside joke that only the two of you share.\n",
      "\n",
      "Example:\n",
      "\n",
      "\"Happy birthday to my favorite friend, Andrew! I can't believe it's been [X] years since we met. You've been there for me through thick and thin, and I'm so grateful for your friendship. Here's to another year of adventures and good times together! 🎉\"\n",
      "\n",
      "2. Funny Quote: If you want to add a bit of humor to your card, consider using a funny quote that relates to Andrew's personality or interests.\n",
      "\n",
      "Example:\n",
      "\n",
      "\"Happy birthday to the most awesome Andrew in the world! May your day be as epic as your beard and your love for [insert hobby or interest here] 😂\"\n",
      "\n",
      "3. Heartfelt Words: If you want to express your feelings in a more heartfelt way, try writing a message that speaks to the importance of Andrew in your life.\n",
      "\n",
      "Example:\n",
      "\n",
      "\"Andrew, you're more than just a friend to me. You're a constant source of support, laughter, and inspiration. I'm so grateful to have you in my life, and I can't wait to see what the next year brings for you. Happy birthday, my dear friend! ❤️\"\n",
      "\n",
      "4. Inside Joke: If you and Andrew share a special inside joke or reference, consider including it in your card. It will make the message more personal and meaningful to him.\n",
      "\n",
      "Example:\n",
      "\n",
      "\"Happy birthday to the only person I know who can make me laugh as hard as I did when we [insert inside joke here]! I hope your day is as amazing as you are, Andrew. 😂\"\n",
      "\n",
      "Remember, the most important thing is to be sincere and genuine in your message. Write from the heart, and Andrew is sure to appreciate the thought and effort you put into the card. Happy birthday, Andrew! 🎉\n"
     ]
    }
   ],
   "source": [
    "# pass prompt to the llama function, store output as 'response' then print\n",
    "response = llama(prompt)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a573ca3f-9ffc-4f08-91eb-7e88d99d2f9f",
   "metadata": {
    "height": 75
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "[INST]Help me write a birthday card for my dear friend Andrew.[/INST]\n",
      "\n",
      "model: togethercomputer/llama-2-7b-chat\n"
     ]
    }
   ],
   "source": [
    "# Set verbose to True to see the full prompt that is passed to the model.\n",
    "prompt = \"Help me write a birthday card for my dear friend Andrew.\"\n",
    "response = llama(prompt, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a0c9bc9-9777-45bb-9e52-1cd99a42e716",
   "metadata": {},
   "source": [
    "### Chat vs. base models\n",
    "\n",
    "Ask model a simple question to demonstrate the different behavior of chat vs. base models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b37cc261-a9d5-486c-a0f1-7f0ce93f403e",
   "metadata": {
    "height": 98
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "[INST]What is the capital of France?[/INST]\n",
      "\n",
      "model: togethercomputer/llama-2-7b-chat\n"
     ]
    }
   ],
   "source": [
    "### chat model\n",
    "prompt = \"What is the capital of France?\"\n",
    "response = llama(prompt, \n",
    "                 verbose=True,\n",
    "                 model=\"togethercomputer/llama-2-7b-chat\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c7d596d0-9469-4220-a03a-b50b0226776d",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  The capital of France is Paris.\n"
     ]
    }
   ],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "71bef6df-ce83-4be7-98a7-cd0c353c4188",
   "metadata": {
    "height": 115
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "What is the capital of France?\n",
      "\n",
      "model: togethercomputer/llama-2-7b\n"
     ]
    }
   ],
   "source": [
    "### base model\n",
    "prompt = \"What is the capital of France?\"\n",
    "response = llama(prompt, \n",
    "                 verbose=True,\n",
    "                 add_inst=False,\n",
    "                 model=\"togethercomputer/llama-2-7b\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21bf01a1-2c3e-4073-a4b7-546f5de0e5a1",
   "metadata": {},
   "source": [
    "Note how the prompt **does not** include the `[INST]` and `[/INST]` tags as `add_inst` was set to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1941be3a-de66-4d08-808e-f93a0393f864",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10. What is the capital of Germany?\n",
      "11. What is the capital of Greece?\n",
      "12. What is the capital of Hungary?\n",
      "13. What is the capital of Iceland?\n",
      "14. What is the capital of India?\n",
      "15. What is the capital of Indonesia?\n",
      "16. What is the capital of Iran?\n",
      "17. What is the capital of Iraq?\n",
      "18. What is the capital of Ireland?\n",
      "19. What is the capital of Israel?\n",
      "20. What is the capital of Italy?\n",
      "21. What is the capital of Japan?\n",
      "22. What is the capital of Jordan?\n",
      "23. What is the capital of Kazakhstan?\n",
      "24. What is the capital of Kenya?\n",
      "25. What is the capital of Kuwait?\n",
      "26. What is the capital of Kyrgyzstan?\n",
      "27. What is the capital of Laos?\n",
      "28. What is the capital of Latvia?\n",
      "29. What is the capital of Lebanon?\n",
      "30. What is the capital of Lesotho?\n",
      "31. What is the capital of Liberia?\n",
      "32. What is the capital of Libya?\n",
      "33. What is the capital of Liechtenstein?\n",
      "34. What is the capital of Lithuania?\n",
      "35. What is the capital of Luxembourg?\n",
      "36. What is the capital of Macedonia?\n",
      "37. What is the capital of Madagascar?\n",
      "38. What is the capital of Malawi?\n",
      "39. What is the capital of Malaysia?\n",
      "40. What is the capital of Maldives?\n",
      "41. What is the capital of Mali?\n",
      "42. What is the capital of Malta?\n",
      "43. What is the capital of Marshall Islands?\n",
      "44. What is the capital of Mauritania?\n",
      "45. What is the capital of Mauritius?\n",
      "46. What is the capital of Mexico?\n",
      "47. What is the capital of Micronesia?\n",
      "48. What is the capital of Moldova?\n",
      "49. What is the capital of Monaco?\n",
      "50. What is the capital of Mongolia?\n",
      "51. What is the capital of Montenegro?\n",
      "52. What is the capital of Morocco?\n",
      "53. What is the capital of Mozambique?\n",
      "54. What is the capital of Myanmar?\n",
      "55. What is the capital of Namibia?\n",
      "56. What is the capital of Nauru?\n",
      "57. What is the capital of Nepal?\n",
      "58. What is the capital of Netherlands?\n",
      "59. What is the capital of New Zealand?\n",
      "60. What is the capital of Nicaragua?\n",
      "61. What is the capital of Niger?\n",
      "62. What is the capital of Nigeria?\n",
      "63. What is the capital of Norway?\n",
      "64. What is the capital of Oman?\n",
      "65. What is the capital of Pakistan?\n",
      "66. What is the capital of Palau?\n",
      "67. What is the capital of Palestine?\n",
      "68. What is the capital of Panama?\n",
      "69. What is the capital of Papua New Guinea?\n",
      "70. What is the capital of Paraguay?\n",
      "71. What is the capital of Peru?\n",
      "72. What is the capital of Philippines?\n",
      "73. What is the capital of Poland?\n",
      "74. What is the capital of Portugal?\n",
      "75. What is the capital of Qatar?\n",
      "76. What is the capital of Romania?\n",
      "77. What is the capital of Russia?\n",
      "78. What is the capital of Rwanda?\n",
      "79. What is the capital of Saint Kitts and Nevis?\n",
      "80. What is the capital of Saint Lucia?\n",
      "81. What is the capital of Saint Vincent and the Grenadines?\n",
      "82. What is the capital of Samoa?\n",
      "83. What is the capital of San Marino?\n",
      "84. What is the capital of Sao Tome and Principe?\n",
      "85. What is the capital of Saudi Arabia?\n",
      "86. What is the capital of Senegal?\n",
      "87. What is the capital of Serbia?\n",
      "88. What is the capital of Seychelles?\n",
      "89. What is the capital of Sierra Leone?\n",
      "90. What is the capital of Singapore?\n",
      "91. What is the capital of Slovakia?\n",
      "92. What is the capital of Sloven\n"
     ]
    }
   ],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e4b6e629",
   "metadata": {
    "height": 149
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "[INST]What is the capital of France?[/INST]\n",
      "\n",
      "model: togethercomputer/llama-2-7b\n",
      "\n",
      "[INST]What is the capital of France?[/INST]\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is the capital of France?[/INST]Paris\n",
      "[INST]What is\n"
     ]
    }
   ],
   "source": [
    "### base model\n",
    "prompt = \"What is the capital of France?\"\n",
    "response = llama(prompt, \n",
    "                 verbose=True,\n",
    "                 add_inst=True,\n",
    "                 model=\"togethercomputer/llama-2-7b\")\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69ac25a-95cd-43da-91e4-2d3c885af811",
   "metadata": {},
   "source": [
    "### Changing the temperature setting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "47c11476-c128-4370-9405-e2899c0284ba",
   "metadata": {
    "height": 194
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Of course! Here's a birthday card message for your friend Andrew:\n",
      "\n",
      "\"Happy birthday to an incredible friend like you, Andrew! 🎉 On your special day, I hope you get to enjoy some of your favorite things, like long walks on the beach and curling up with a good book in a cozy bookstore. 📚🌊\n",
      "\n",
      "I'm so grateful for your love of learning and your passion for sharing your knowledge with others. Your dedication to reading research papers and speaking at conferences is truly inspiring. 💡🎤\n",
      "\n",
      "And let's not forget your love for pandas! 🐼 They're such adorable and fascinating creatures, just like you. 😊\n",
      "\n",
      "Here's to another amazing year of adventures, learning, and friendship! Cheers, Andrew! 🥳🎂\"\n"
     ]
    }
   ],
   "source": [
    "prompt = \"\"\"\n",
    "Help me write a birthday card for my dear friend Andrew.\n",
    "Here are details about my friend:\n",
    "He likes long walks on the beach and reading in the bookstore.\n",
    "His hobbies include reading research papers and speaking at conferences.\n",
    "His favorite color is light blue.\n",
    "He likes pandas.\n",
    "\"\"\"\n",
    "response = llama(prompt, temperature=0.0)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "065d446f-2eef-4f34-858b-c5146e154cf0",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Of course! Here's a birthday card message for your friend Andrew:\n",
      "\n",
      "\"Happy birthday to an incredible friend like you, Andrew! 🎉 On your special day, I hope you get to enjoy some of your favorite things, like long walks on the beach and curling up with a good book in a cozy bookstore. 📚🌊\n",
      "\n",
      "I'm so grateful for your love of learning and your passion for sharing your knowledge with others. Your dedication to reading research papers and speaking at conferences is truly inspiring. 💡🎤\n",
      "\n",
      "And let's not forget your love for pandas! 🐼 They're such adorable and fascinating creatures, just like you. 😊\n",
      "\n",
      "Here's to another amazing year of adventures, learning, and friendship! Cheers, Andrew! 🥳🎂\"\n"
     ]
    }
   ],
   "source": [
    "# Run the code again - the output should be identical\n",
    "response = llama(prompt, temperature=0.0)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bb83e70c-c629-470a-8013-cc63c56b5870",
   "metadata": {
    "height": 194
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Of course! Here's a sample birthday card message for your friend Andrew:\n",
      "\n",
      "\"Happy birthday, Andrew! 🎉 I hope your day is as amazing as you are. 😊 I'm so grateful for our long walks on the beach and our late-night conversations about the latest research papers. Your passion for learning and sharing your knowledge with the world never ceases to inspire me. 📚💡 And let's be real, who doesn't love a good panda? 🐼 Wishing you a year filled with light blue skies, fascinating reads, and plenty of panda cuddles. Happy birthday, my dear friend! 🎂🎁\"\n"
     ]
    }
   ],
   "source": [
    "prompt = \"\"\"\n",
    "Help me write a birthday card for my dear friend Andrew.\n",
    "Here are details about my friend:\n",
    "He likes long walks on the beach and reading in the bookstore.\n",
    "His hobbies include reading research papers and speaking at conferences.\n",
    "His favorite color is light blue.\n",
    "He likes pandas.\n",
    "\"\"\"\n",
    "response = llama(prompt, temperature=0.9)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6834c2db-16c7-46ed-8c79-9aac6041fe66",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Of course, I'd be happy to help you write a birthday card for your friend Andrew! Here's a suggestion:\n",
      "\n",
      "Dear Andrew,\n",
      "\n",
      "Happy birthday to our dear friend, the walking-talking embodiment of intellectual curiosity! 📚🏖️\n",
      "\n",
      "As you celebrate another trip around the sun, we hope you take some time to indulge in your favorite hobby: reading research papers and speaking at conferences. 🤔📝 We're pretty sure you're the only person who can make a lecture on quantum physics sound like a pages-turning thriller. 🎤\n",
      "\n",
      "But enough about work (for now). Have you ever considered combining your two loves and writing a groundbreaking paper on the psychological effects of reading on the beach? 🏖️🧠 Just a thought.\n",
      "\n",
      "In all seriousness, Andrew, we're so grateful for your friendship and your infectious enthusiasm for learning. May your birthday be as enjoyable as a long walk on a sunny day, with a side of light blue sky to match your favorite color. 🌞💙 And if you happen to see any pandas on your Traverse City adventures, do give them a belly rub from us. 🐼\n",
      "\n",
      "Happy birthday, dear Andrew! 🎉\n",
      "\n",
      "Warm regards,\n",
      "[Your Name]\n"
     ]
    }
   ],
   "source": [
    "# run the code again - the output should be different\n",
    "response = llama(prompt, temperature=0.9)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "295682fa-ca26-4924-89ca-a86710e64f78",
   "metadata": {},
   "source": [
    "### Changing the max tokens setting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ede4dfa8-0d2c-42da-b588-701119bd136f",
   "metadata": {
    "height": 194
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Of course! Here's a birthday card message for your friend Andrew:\n",
      "\n",
      "\"\n"
     ]
    }
   ],
   "source": [
    "prompt = \"\"\"\n",
    "Help me write a birthday card for my dear friend Andrew.\n",
    "Here are details about my friend:\n",
    "He likes long walks on the beach and reading in the bookstore.\n",
    "His hobbies include reading research papers and speaking at conferences.\n",
    "His favorite color is light blue.\n",
    "He likes pandas.\n",
    "\"\"\"\n",
    "response = llama(prompt,max_tokens=20)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5333a590-b54a-44f3-a627-1db0fe462315",
   "metadata": {},
   "source": [
    "The next cell reads in the text of the children's book *The Velveteen Rabbit* by Margery Williams, and stores it as a string named `text`. (Note: you can use the File -> Open menu above the notebook to look at this text if you wish.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8b0a6d79-8c32-4b56-869d-ec7b4b9e526c",
   "metadata": {
    "height": 58
   },
   "outputs": [],
   "source": [
    "with open(\"TheVelveteenRabbit.txt\", \"r\", encoding='utf=8') as file:\n",
    "    text = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "25e31bf5-b323-4771-ae59-70b73e00ec4b",
   "metadata": {
    "height": 98
   },
   "outputs": [],
   "source": [
    "prompt = f\"\"\"\n",
    "Give me a summary of the following text in 50 words:\\n\\n\n",
    "{text}\n",
    "\"\"\"\n",
    "response = llama(prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7f15ff3c-0068-4abe-8e6f-e3baf92dd31d",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'error': {'message': 'Input validation error: `inputs` tokens + `max_new_tokens` must be <= 4097. Given: 3974 `inputs` tokens and 1024 `max_new_tokens`', 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': None}}\n"
     ]
    }
   ],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83d0f2dc-6822-4636-92cd-d7cd77d9a27f",
   "metadata": {},
   "source": [
    "Running the cell above returns an error because we have too many tokens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "04efbc3e-e67f-4300-a46d-7bf3834077dc",
   "metadata": {
    "height": 58
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4998"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sum of input tokens (prompt + Velveteen Rabbit text) and output tokens\n",
    "3974 + 1024"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43ed7ac3-24df-48da-8868-5cddd6176dfe",
   "metadata": {},
   "source": [
    "For Llama 2 chat models, the sum of the input and max_new_tokens parameter must be <= 4097 tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ee7e1c8d-b16c-41af-8c83-79ce559f860a",
   "metadata": {
    "height": 58
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "123"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate tokens available for response after accounting for 3974 input tokens\n",
    "4097 - 3974"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e93887be-c3ea-44d2-86fc-a4aa9e101a37",
   "metadata": {
    "height": 143
   },
   "outputs": [],
   "source": [
    "# set max_tokens to stay within limit on input + output tokens\n",
    "prompt = f\"\"\"\n",
    "Give me a summary of the following text in 50 words:\\n\\n\n",
    "{text}\n",
    "\"\"\"\n",
    "response = llama(prompt,\n",
    "                max_tokens=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b9e8b6db-e1ac-4480-bd86-d57334d6db57",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  The Velveteen Rabbit is a heartwarming story about the relationship between a young boy and his stuffed toy rabbit. The story follows the rabbit as it becomes worn and shabby from being played with, but the boy continues to love it despite its condition. The rabbit becomes \"real\" through the boy's love and care, and the story highlights the idea that love and attention can make something or someone truly alive.\n",
      "\n",
      "The story is written in a simple and straightforward style, making it easy to follow and understand. The use of descriptive language\n"
     ]
    }
   ],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "3ccbe1ce-35f6-4bdb-888a-5232b3a23794",
   "metadata": {
    "height": 132
   },
   "outputs": [],
   "source": [
    "# increase max_tokens beyond limit on input + output tokens\n",
    "prompt = f\"\"\"\n",
    "Give me a summary of the following text in 50 words:\\n\\n\n",
    "{text}\n",
    "\"\"\"\n",
    "response = llama(prompt,\n",
    "                max_tokens=124)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "02acab62",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'error': {'message': 'Input validation error: `inputs` tokens + `max_new_tokens` must be <= 4097. Given: 3974 `inputs` tokens and 124 `max_new_tokens`', 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': None}}\n"
     ]
    }
   ],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e768b69e-ee6c-445f-b68b-a7480a76a019",
   "metadata": {},
   "source": [
    "### Asking a follow up question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b613eb4a-c922-4037-ad1e-ffb8cdea1c1f",
   "metadata": {
    "height": 194
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Of course! Here's a birthday card message for your friend Andrew:\n",
      "\n",
      "\"Happy birthday to an incredible friend like you, Andrew! 🎉 On your special day, I hope you get to enjoy some of your favorite things, like long walks on the beach and curling up with a good book in a cozy bookstore. 📚🌊\n",
      "\n",
      "I'm so grateful for your love of learning and your passion for sharing your knowledge with others. Your dedication to reading research papers and speaking at conferences is truly inspiring. 💡🎤\n",
      "\n",
      "And let's not forget your love for pandas! 🐼 They're such adorable and fascinating creatures, just like you. 😊\n",
      "\n",
      "Here's to another amazing year of adventures, learning, and friendship! Cheers, Andrew! 🥳🎂\"\n"
     ]
    }
   ],
   "source": [
    "prompt = \"\"\"\n",
    "Help me write a birthday card for my dear friend Andrew.\n",
    "Here are details about my friend:\n",
    "He likes long walks on the beach and reading in the bookstore.\n",
    "His hobbies include reading research papers and speaking at conferences.\n",
    "His favorite color is light blue.\n",
    "He likes pandas.\n",
    "\"\"\"\n",
    "response = llama(prompt)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "4333b1d1-e4fe-436a-a3af-d23eed09b866",
   "metadata": {
    "height": 109
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Of course! Here's a revised version of the paragraph that includes the fact that the person also enjoys teaching:\n",
      "\n",
      "\"John is a highly skilled and experienced software engineer with a passion for programming. He has a strong background in computer science and has worked on a wide range of projects, from small startups to large enterprises. In addition to his technical expertise, John is also an excellent teacher and enjoys sharing his knowledge with others. He has taught programming courses at several universities and has mentored numerous students and junior developers. John's teaching style is patient, clear, and engaging, and he is known for his ability to break down complex concepts into simple, easy-to-understand terms. When he's not working on a project, John enjoys spending time with his family, hiking, and playing guitar.\"\n"
     ]
    }
   ],
   "source": [
    "prompt_2 = \"\"\"\n",
    "Oh, he also likes teaching. Can you rewrite it to include that?\n",
    "\"\"\"\n",
    "response_2 = llama(prompt_2)\n",
    "print(response_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1f8355e-2da8-411d-8bf1-c0b545ddc4da",
   "metadata": {},
   "source": [
    "### (Optional): Using Llama-2 on your own computer!\n",
    "- Llama-2 is free to download on your own machine!\n",
    "- One way to install and use llama on your computer is to go to https://ollama.com/ and download app. It will be like installing a regular application.\n",
    "- To use llama-2, the full instructions are here: https://ollama.com/library/llama2\n",
    "\n",
    "#### Here's an quick summary of how to get started:\n",
    "  - Follow the installation instructions (for Windows, Mac or Linux).\n",
    "  - Open the command line interface (CLI) and type `ollama run llama2`.\n",
    "  - The first time you do this, it will take some time to download the llama-2 model. After that, you'll see \n",
    "> `>>> Send a message (/? for help)`\n",
    "\n",
    "- You can type your prompt and the llama-2 model on your computer will give you a response!\n",
    "- To exit, type `/bye`.\n",
    "- For a list of other commands, type `/?`.\n",
    "\n",
    "![](ollama_example.png \"\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71958a71-89d9-4ccb-88e2-97fa173bb09e",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}