09-BingChatClone.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "66ab3cc5-aee4-415a-9391-1e5d37ccaf1d",
   "metadata": {},
   "source": [
    "# Skill 4: Internet and Websites Search using Bing API - Bing Chat Clone"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306fc0a9-4044-441d-9ba7-f54f32e6ea9f",
   "metadata": {},
   "source": [
    "In this notebook, we'll delve into the ways in which you can **boost your GPT Smart Search Engine with web search functionalities**, utilizing both Langchain and the Azure Bing Search API service.\n",
    "\n",
    "As previously discussed in our other notebooks, **harnessing agents and tools is an effective approach**. We aim to leverage the capabilities of OpenAI's large language models (LLM), such as GPT-3.5 and its successors, to perform the heavy lifting of reasoning and researching on our behalf.\n",
    "\n",
    "There are numerous instances where it is necessary for our Smart Search Engine to have internet access. For instance, we may wish to **enrich an answer with information available on the web**, or **provide users with up-to-date and recent information**, or **finding information on an specific public website**. Regardless of the scenario, we require our engine to base its responses on search results.\n",
    "\n",
    "By the conclusion of this notebook, you'll have a solid understanding of the Bing Search API basics, including **how to create a Web Search Agent using the Bing Search API**, and how these tools can strengthen your chatbot. Additionally, you'll learn about Callbacks, another way  of **how to observe Agent Actions and their significance in bot applications**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c1fb79a3-4856-4721-988c-112813690a90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import requests\n",
    "from typing import Dict, List, Optional, Type\n",
    "import asyncio\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "from langchain import hub\n",
    "from langchain.callbacks.manager import AsyncCallbackManagerForToolRun, CallbackManagerForToolRun\n",
    "from langchain.pydantic_v1 import BaseModel, Field\n",
    "from langchain.tools import BaseTool, StructuredTool, tool\n",
    "from langchain_openai import AzureChatOpenAI\n",
    "from langchain.agents import AgentExecutor, Tool, create_openai_tools_agent\n",
    "from langchain.callbacks.manager import CallbackManager\n",
    "from langchain.agents import initialize_agent, AgentType\n",
    "from langchain.utilities import BingSearchAPIWrapper\n",
    "\n",
    "from common.callbacks import StdOutCallbackHandler\n",
    "from common.prompts import BINGSEARCH_PROMPT\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string.replace(\"$\",\"USD \")))\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "258a6e99-2d4f-4147-b8ee-c64c85296181",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d85dc4b5-2a3e-45bf-bc6c-7f4605149382",
   "metadata": {},
   "source": [
    "## Introduction to Callback Handlers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "108f80d1-9333-4bad-b73a-615429cf9336",
   "metadata": {},
   "source": [
    "**Callbacks**:\n",
    "\n",
    "LangChain provides a callbacks system, another way to monitor/observe agent actions, that allows you to hook into the various stages of your LLM application. This is useful for logging, monitoring, streaming, and other tasks. You can subscribe to these events by using the callbacks argument available throughout the API. This argument is a list of handler objects.\n",
    "\n",
    "**Callback handlers**:\n",
    "\n",
    "CallbackHandlers are objects that implement the CallbackHandler interface, which has a method for each event that can be subscribed to. The CallbackManager will call the appropriate method on each handler when the event is triggered.\n",
    "\n",
    "---\n",
    "\n",
    "We will incorporate a handler for the callbacks, enabling us to observe the response as it streams and to gain insights into the Agent's reasoning process. This will prove incredibly valuable when we aim to stream the bot's responses to users and keep them informed about the ongoing process as they await the answer.\n",
    "\n",
    "Our custom handler is in the folder `common/callbacks.py`. Go and take a look at it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1380fb0d-3502-4fc0-b729-9bc46d5c9804",
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_handler = StdOutCallbackHandler()\n",
    "cb_manager = CallbackManager(handlers=[cb_handler])\n",
    "\n",
    "COMPLETION_TOKENS = 2000\n",
    "\n",
    "llm = AzureChatOpenAI(deployment_name=os.environ[\"GPT35_DEPLOYMENT_NAME\"], \n",
    "                      temperature=0.5, max_tokens=COMPLETION_TOKENS, \n",
    "                      streaming=True, callback_manager=cb_manager)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11da70c2-60b6-47fb-94f1-aa11291fa40c",
   "metadata": {},
   "source": [
    "### Creating a custom tool - Bing Search API tool"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc30c9d-605d-4ada-9358-f926aeed2e48",
   "metadata": {},
   "source": [
    "Langhain has already a pre-created tool called BingSearchAPIWrapper, however we are going to make it a bit better by using the results function instead of the run function, that way we not only have the text results, but also the title and link(source) of each snippet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d3d155ae-16eb-458a-b2ed-5aa9a9b84ed8",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SearchInput(BaseModel):\n",
    "    query: str = Field(description=\"should be a search query\")\n",
    "\n",
    "class MyBingSearch(BaseTool):\n",
    "    \"\"\"Tool for a Bing Search Wrapper\"\"\"\n",
    "    \n",
    "    name = \"Searcher\"\n",
    "    description = \"useful to search the internet.\\n\"\n",
    "    args_schema: Type[BaseModel] = SearchInput\n",
    "\n",
    "    k: int = 5\n",
    "    \n",
    "    def _run(self, query: str, run_manager: Optional[CallbackManagerForToolRun] = None) -> str:\n",
    "        bing = BingSearchAPIWrapper(k=self.k)\n",
    "        return bing.results(query,num_results=self.k)\n",
    "            \n",
    "    async def _arun(self, query: str, run_manager: Optional[AsyncCallbackManagerForToolRun] = None) -> str:\n",
    "        bing = BingSearchAPIWrapper(k=self.k)\n",
    "        loop = asyncio.get_event_loop()\n",
    "        results = await loop.run_in_executor(ThreadPoolExecutor(), bing.results, query, self.k)\n",
    "        return results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e3da24-5fa5-47c3-ab43-1840e05f1f54",
   "metadata": {},
   "source": [
    "### Creating another custom tool - WebFetcher: Visits a website and extracts the text\n",
    "    You will need GPT-4 with a big context token size for this tool since the content of a website can be very lenghty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "74b714b6-e6e6-492b-8434-6081a1ff183e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_html(content) -> str:\n",
    "    soup = BeautifulSoup(content, 'html.parser')\n",
    "    text_content_with_links = soup.get_text()\n",
    "    return text_content_with_links\n",
    "\n",
    "def fetch_web_page(url: str) -> str:\n",
    "    HEADERS = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:90.0) Gecko/20100101 Firefox/90.0'}\n",
    "    response = requests.get(url, headers=HEADERS)\n",
    "    return parse_html(response.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "201d1f47-519e-40b3-80b2-2d29a494dd62",
   "metadata": {},
   "outputs": [],
   "source": [
    "web_fetch_tool = Tool.from_function(\n",
    "    func=fetch_web_page,\n",
    "    name=\"WebFetcher\",\n",
    "    description=\"useful to fetch the content of a url\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c35b2f7-fd32-4ab0-bdd4-966da6e9587d",
   "metadata": {},
   "source": [
    "### Creating the Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a3d6569-0c61-4b1c-9263-431304577551",
   "metadata": {},
   "source": [
    "Now, we create our OpenAI Tools type agent that uses our custom tools and our custom prompt `BING_PROMPT_PREFIX`. Check it out in `prompts.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2c6cf721-76bb-47b6-aeeb-9ff4ff92b1f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tools = [MyBingSearch(k=5), web_fetch_tool] # With GPT-4 you can add the web_fetch_tool\n",
    "\n",
    "tools = [MyBingSearch(k=5)] # With GPT-3.5 \n",
    "\n",
    "prompt = BINGSEARCH_PROMPT\n",
    "\n",
    "# Construct the OpenAI Tools agent\n",
    "agent = create_openai_tools_agent(llm, tools, prompt)\n",
    "\n",
    "# Create an agent executor by passing in the agent and tools\n",
    "agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=False, \n",
    "                               return_intermediate_steps=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4bf377b4-0913-4695-bcf9-31629b2cf66f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[MyBingSearch()]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent_executor.tools"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7232260e-e972-4288-b0b5-0b605e584528",
   "metadata": {},
   "source": [
    "Try some of the below questions, or others that you might like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fa949cea-c9aa-4529-a75f-61084ffffd7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# QUESTION = \"Create a list with the main facts on What is happening with the oil supply in the world right now?\"\n",
    "# QUESTION = \"How much is 50 USD in Euros and is it enough for an average hotel in Madrid?\"\n",
    "# QUESTION = \"My son needs to build a pinewood car for a pinewood derbi, how do I build such a car?\"\n",
    "# QUESTION = \"I'm planning a vacation to Greece, tell me budget for a family of 4, in Summer, for 7 days including travel, lodging and food costs\"\n",
    "# QUESTION = \"Who won the 2023 superbowl and who was the MVP?\"\n",
    "QUESTION = \"\"\"\n",
    "compare the number of job opennings (provide the exact number), the average salary within 15 miles of Dallas, TX, for these ocupations:\n",
    "\n",
    "- ADN Registerd Nurse \n",
    "- Occupational therapist assistant\n",
    "- Dental Hygienist\n",
    "- Graphic Designer\n",
    "\n",
    "\n",
    "# Create a table with your findings. Place the sources on each cell.\n",
    "# \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e8e0b32-a6b5-4b1c-943d-e57b737213fa",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Agent Actions/Observations during streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "003327ac-2851-48ef-8a6b-2d8c2004bb2e",
   "metadata": {},
   "source": [
    "Streaming is an important UX consideration for LLM apps, and agents are no exception. Streaming with agents is made more complicated by the fact that it’s not just tokens of the final answer that you will want to stream, but you may also want to stream back the intermediate steps an agent takes.\n",
    "\n",
    "The outputs also contain richer structured information inside of actions and steps, which could be useful in some situations, but can also be harder to parse.\n",
    "\n",
    "At the end of Notebook 3 we learned that streaming can be simply achieve by doing this:\n",
    "\n",
    "```python\n",
    "for chunk in chain.stream({\"question\": QUESTION, \"language\": \"English\", \"history\":\"\"}):\n",
    "    print(chunk, end=\"\", flush=True)\n",
    "```\n",
    "\n",
    "At the end of Notebook 6 we learned about the new astream_events API (beta).\n",
    "\n",
    "```python\n",
    "async for event in agent_with_chat_history.astream_events(\n",
    "    {\"question\": QUESTION}, config=config, version=\"v1\"):\n",
    "```\n",
    "\n",
    "Now we are going to achieve the same result of the astream_events API, by combining Callbacks with the astream() function:\n",
    "\n",
    "    With Agents, we would need to parse the information contained on each streamed chunk since it contains a lot of information and also use the callback handler to stream the tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ca948d67-6717-4843-b7ab-b13155aa8581",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calling Tool: `Searcher` with input `{'query': 'job openings for ADN Registered Nurse within 15 miles of Dallas, TX'}`\n",
      "---\n",
      "Calling Tool: `Searcher` with input `{'query': 'average salary for ADN Registered Nurse within 15 miles of Dallas, TX'}`\n",
      "---\n",
      "Calling Tool: `Searcher` with input `{'query': 'job openings for Occupational therapist assistant within 15 miles of Dallas, TX'}`\n",
      "---\n",
      "Calling Tool: `Searcher` with input `{'query': 'average salary for Occupational therapist assistant within 15 miles of Dallas, TX'}`\n",
      "---\n",
      "Calling Tool: `Searcher` with input `{'query': 'job openings for Dental Hygienist within 15 miles of Dallas, TX'}`\n",
      "---\n",
      "Calling Tool: `Searcher` with input `{'query': 'average salary for Dental Hygienist within 15 miles of Dallas, TX'}`\n",
      "---\n",
      "Calling Tool: `Searcher` with input `{'query': 'job openings for Graphic Designer within 15 miles of Dallas, TX'}`\n",
      "---\n",
      "Calling Tool: `Searcher` with input `{'query': 'average salary for Graphic Designer within 15 miles of Dallas, TX'}`\n",
      "---\n",
      "I have gathered the information on job openings and average salaries within 15 miles of Dallas, TX for the specified occupations. Here's a table with the findings:\n",
      "\n",
      "| Occupation                    | Job Openings | Average Salary | Sources                                                                                       |\n",
      "|-------------------------------|--------------|-----------------|-----------------------------------------------------------------------------------------------|\n",
      "| ADN Registered Nurse          | 369          | $35.52 per hour | [Indeed](https://www.indeed.com/q-RN-Adn-l-Dallas,-TX-jobs.html), [ZipRecruiter](https://www.ziprecruiter.com/Salaries/ADN-Nurse-Salary-in-Dallas,TX) |\n",
      "| Occupational Therapist Asst.  | 157          | $77,826 per year| [Indeed](https://www.indeed.com/q-occupational-therapist-assistanc-l-dallas,-tx-jobs.html), [Glassdoor](https://www.glassdoor.com/Salaries/dallas-occupational-therapy-assistant-salary-SRCH_IL.0,6_IM218_KO7,37.htm) |\n",
      "| Dental Hygienist              | 93           | $49.48 per hour | [Indeed](https://www.indeed.com/q-Dental-Hygienist-l-Dallas,-TX-jobs.html), [Glassdoor](https://www.glassdoor.com/Salaries/dallas-dental-hygienist-salary-SRCH_IL.0,6_IM218_KO7,23.htm) |\n",
      "| Graphic Designer              | 57           | $56,006 per year| [Indeed](https://www.indeed.com/q-Graphic-Designer-l-Dallas-Fort-Worth,-TX-jobs.html), [Glassdoor](https://www.glassdoor.com/Salaries/graphic-designer-salary-SRCH_IM218_KO0,16.htm) |\n",
      "\n",
      "The sources for each cell are provided for transparency and reference.\n",
      "\n",
      "If you need further assistance or additional information, feel free to ask!"
     ]
    }
   ],
   "source": [
    "async for chunk in agent_executor.astream({\"question\": QUESTION}):\n",
    "    # Agent Action\n",
    "    if \"actions\" in chunk:\n",
    "        for action in chunk[\"actions\"]:\n",
    "            print(f\"Calling Tool: `{action.tool}` with input `{action.tool_input}`\")\n",
    "    # Observation\n",
    "    elif \"steps\" in chunk:\n",
    "        # Uncomment if you need to have the information retrieve from the tool\n",
    "        # for step in chunk[\"steps\"]:\n",
    "        #     print(f\"Tool Result: `{step.observation}`\")\n",
    "        continue\n",
    "    # Final result\n",
    "    elif \"output\" in chunk:\n",
    "        # No need to print the final output again since we would be streaming it as it is produced\n",
    "        # print(f'Final Output: {chunk[\"output\"]}') \n",
    "        continue\n",
    "    else:\n",
    "        raise ValueError()\n",
    "    print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25d49f6e-73eb-4513-9341-fcad5e5e33e6",
   "metadata": {},
   "source": [
    "#### Without showing the intermedite steps, just the final answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ca910f71-60fb-4758-b4a9-757e37eb421f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Based on the current exchange rate, 50 USD is approximately 45 EUR [[1]](https://www.xe.com/en/currencyconverter/convert/?Amount=50&From=USD&To=EUR). \n",
      "\n",
      "As for the average hotel cost in Madrid, the average price for a hotel for one week in Madrid is approximately $625, which is around 565 EUR. This means that 50 USD is not enough to cover the average hotel cost for one week in Madrid [[2]](https://www.budgetyourtrip.com/hotels/spain/madrid-3117735).\n",
      "\n",
      "If you have any other questions or need further assistance, feel free to ask!"
     ]
    }
   ],
   "source": [
    "QUESTION = \"How much is 50 USD in Euros and is it enough for an average hotel in Madrid?\"\n",
    "\n",
    "try:\n",
    "    response = agent_executor.invoke({\"question\":QUESTION})\n",
    "except Exception as e:\n",
    "    response = str(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "25a410b2-9950-43f5-8f14-b333bdc24ff2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Based on the current exchange rate, 50 USD is approximately 45 EUR [[1]](https://www.xe.com/en/currencyconverter/convert/?Amount=50&From=USD&To=EUR). \n",
       "\n",
       "As for the average hotel cost in Madrid, the average price for a hotel for one week in Madrid is approximately USD 625, which is around 565 EUR. This means that 50 USD is not enough to cover the average hotel cost for one week in Madrid [[2]](https://www.budgetyourtrip.com/hotels/spain/madrid-3117735).\n",
       "\n",
       "If you have any other questions or need further assistance, feel free to ask!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(response[\"output\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9ba3e35-8021-4262-8494-d1aee3862f7e",
   "metadata": {},
   "source": [
    "## QnA to specific websites\n",
    "\n",
    "There are several use cases where we want the smart bot to answer questions about a specific company's public website. There are two approaches we can take:\n",
    "\n",
    "1. Create a crawler script that runs regularly, finds every page on the website, and pushes the documents to Azure Cognitive Search.\n",
    "2. Since Bing has likely already indexed the public website, we can utilize Bing search targeted specifically to that site, rather than attempting to index the site ourselves and duplicate the work already done by Bing's crawler.\n",
    "\n",
    "Below are some sample questions related to specific sites. Take a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e925ee4a-d295-4815-9e8c-bd6999f48892",
   "metadata": {},
   "outputs": [],
   "source": [
    "# QUESTION = \"information on how to kill wasps in homedepot.com\"\n",
    "QUESTION = \"in target.com, find how what's the price of a Nesspresso coffee machine and of a Keurig coffee machine\"\n",
    "# QUESTION = \"in microsoft.com, find out what is the latests news on quantum computing\"\n",
    "# QUESTION = \"give me on a list the main points on the latest investor report from mondelezinternational.com\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1f7c4e6d-03a8-47f8-b859-f7b397981a6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calling Tool: `Searcher` with input `{'query': 'Nesspresso coffee machine price site:target.com'}`\n",
      "---\n",
      "Calling Tool: `Searcher` with input `{'query': 'Keurig coffee machine price site:target.com'}`\n",
      "---\n",
      "The price of a Nespresso coffee machine and a Keurig coffee machine at Target.com are as follows:\n",
      "\n",
      "### Nespresso Coffee Machines\n",
      "1. **Nespresso Vertuo Next Coffee Maker and Espresso Machine by De'Longhi**:\n",
      "   - Price: $139.99 [[1]](https://www.target.com/p/nespresso-vertuo-next-coffee-maker-and-espresso-machine-by-delonghi-gray/-/A-79332989)\n",
      "\n",
      "2. **Nespresso VertuoPlus Deluxe Coffee Maker and Espresso Machine by De'Longhi**:\n",
      "   - Price: $139.99 (price in cart) [[2]](https://www.target.com/p/nespresso-vertuo-plus-deluxe-coffee-maker-and-espresso-machine-by-delonghi-titan/-/A-82749274)\n",
      "\n",
      "### Keurig Coffee Machines\n",
      "1. **Keurig K-Supreme Plus Coffee Maker - Stainless Steel**:\n",
      "   - Price: $219.99 [[3]](https://www.target.com/b/keurig/-/N-5uvm4)\n",
      "\n",
      "2. **Keurig K-Iced Plus Single-Serve K-Cup Pod Coffee Maker with Iced Coffee Option**:\n",
      "   - Price: $129.99 [[4]](https://www.target.com/s/cheapest+keurig+coffee+makers)\n",
      "\n",
      "These prices provide an overview of the cost of Nespresso and Keurig coffee machines available at Target.com."
     ]
    }
   ],
   "source": [
    "async for chunk in agent_executor.astream({\"question\": QUESTION}):\n",
    "    # Agent Action\n",
    "    if \"actions\" in chunk:\n",
    "        for action in chunk[\"actions\"]:\n",
    "            print(f\"Calling Tool: `{action.tool}` with input `{action.tool_input}`\")\n",
    "    # Observation\n",
    "    elif \"steps\" in chunk:\n",
    "        # Uncomment if you need to have the information retrieve from the tool\n",
    "        # for step in chunk[\"steps\"]:\n",
    "        #     print(f\"Tool Result: `{step.observation}`\")\n",
    "        continue\n",
    "    # Final result\n",
    "    elif \"output\" in chunk:\n",
    "        # No need to print the final output again since we would be streaming it as it is produced\n",
    "        # print(f'Final Output: {chunk[\"output\"]}') \n",
    "        continue\n",
    "    else:\n",
    "        raise ValueError()\n",
    "    print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56cbc405-26e2-471e-9626-2a0df07f5ddc",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7381ea5f-7269-4e1f-8b0c-1e2c04bd84c0",
   "metadata": {},
   "source": [
    "In this notebook, we learned how to create a Bing Chat clone using a clever prompt with specific search and formatting instructions. We also learned about combining the Callback Handlers with the agent stream() or astream() functions, to stream the response from the LLM while showing the intermediate steps.  \n",
    "\n",
    "The outcome is an agent capable of conducting intelligent web searches and performing research on our behalf. This agent provides us with answers to our questions along with appropriate URL citations and links!\n",
    "\n",
    "**Note**: as we have said before GPT-4 will be more accurate following instructions, hold more space for context, and provide better responses."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02073623-91b4-40d6-8eaf-cb6d9c6a7a9a",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "\n",
    "The Next Notebook will guide you on how we stick everything together. How do we use the features of all notebooks and create a brain agent that can respond to any request accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44adf1ce-cb34-41dc-a7bc-ab73d64a5dc3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}