14-LangServe-API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3dd1de00-d697-4b2f-a656-60177046d24d",
   "metadata": {},
   "source": [
    "# Building the FastAPI Backend using Langserve"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f15266ef-9986-4e04-9cc8-a0195979af99",
   "metadata": {},
   "source": [
    "Let's first review what we have accomplished so far in deploying our Smart Bot:\n",
    "\n",
    "1) **Notebook 12**: Instructions on deploying a Backend API using the Azure Bot Service.\n",
    "2) **Notebook 13**: Guidelines on interfacing with the Bot Service programmatically using POST requests.\n",
    "\n",
    "Here are the pros and cons of using the Bot Service:\n",
    "\n",
    "**Pros**:\n",
    "- Easily connects to multiple channels, including O365 emails, MS Teams, web chat plugins, etc.\n",
    "- The Bot Framework Python SDKs provide numerous utilities like typing indicators, proactive messages, cards, file uploads, etc.\n",
    "- Includes built-in authentication and logging mechanisms, requiring minimal effort from us.\n",
    "- Offers SDKs for Python, JavaScript, and .NET.\n",
    "- Enables easy integration with the Application Insights Service for application monitoring.\n",
    "- Like other Microsoft services, it is backed by the Microsoft product and support teams.\n",
    "\n",
    "**Cons**:\n",
    "- Does not yet support streaming.\n",
    "- Lacks support for private endpoints.\n",
    "- As a service, it cannot be containerized or run on Kubernetes, container apps, etc.\n",
    "- Requires a steeper learning curve to fully understand all its capabilities.\n",
    "\n",
    "As an alternative, in this notebook, we will build another Backend API using FastAPI with LangServe. <br>This API is self-contained, allowing it to be packaged in a Docker container and deployed anywhere. \n",
    "\n",
    "In this notebook, we will zip the code and upload it to a new slot in the same Azure Web App service where the BotService API resides.\n",
    "\n",
    "\n",
    "From the [LANGSERVE DOCUMENTATION](https://python.langchain.com/docs/langserve):\n",
    "\n",
    "    LangServe helps developers deploy LangChain runnables and chains as a REST API.\n",
    "\n",
    "    This library is integrated with FastAPI and uses pydantic for data validation.\n",
    "\n",
    "    In addition, it provides a client that can be used to call into runnables deployed on a server. A JavaScript client is available in LangChain.js."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d11aef69-49e3-4ad3-b73e-bb6d784b7269",
   "metadata": {},
   "source": [
    "## The main file: Server.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e185ba88-2a67-42e2-9f46-4aeaf3ba1e5f",
   "metadata": {},
   "source": [
    "Just as the main code in the Bot Service API resides in bot.py, in this FastAPI backend, the main code resides in `apps/backend/langserve/app/server.py`\n",
    "\n",
    "**Take a look at it!**\n",
    "\n",
    "In `server.py` you will see that we created 4 endpoints:\n",
    "\n",
    "- `/docs/` \n",
    "  - This endpoint shows the OpenAPI definition (Swagger) of the API\n",
    "- `/chatgpt/`\n",
    "  - This endpoint uses a simple LLM to answer with no system prompt\n",
    "- `/joke/`\n",
    "  - This endpoint uses chain with a LLM + prompt + a custom structured json output (adds the timestamp of the server)\n",
    "- `/agent/`\n",
    "  - This is our the endpoint for our SMART GPT Bot brain agent \n",
    "  \n",
    "For every endpoint all these routes are available: `/invoke/`, `/batch/`, `/stream/` and `/stream_events/`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deb6dbad-7506-45ce-85ad-ac5a42d297c6",
   "metadata": {},
   "source": [
    "## Deploy in Azure App service"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc98f528-707b-4a39-9a0b-3bc76ecd74a1",
   "metadata": {},
   "source": [
    "In `apps/backend/langserve/README.md` you will find all the instructions on how to Zip the code and upload it to the Azure Web App. We will be using the same Azure Web App Service created for the Bot Service API.\n",
    "\n",
    "=> **GO AHEAD NOW AND FOLLOW THE INSTRUCTIONS in `apps/backend/langserve/README.md`**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f725b30-7c90-4a44-9dee-30ccf2f19a97",
   "metadata": {},
   "source": [
    "## (optional) Deploy the server locally"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "986bd37b-53ca-4a66-a6d4-ca9d9c44b16d",
   "metadata": {},
   "source": [
    "1) Go to the file `apps/backend/langserve/app/server.py` and uncomment the following code to test locally:\n",
    "```python\n",
    "    ### uncomment this section to run server in local host #########\n",
    "\n",
    "    # from pathlib import Path\n",
    "    # from dotenv import load_dotenv\n",
    "    # # Calculate the path three directories above the current script\n",
    "    # library_path = Path(__file__).resolve().parents[4]\n",
    "    # sys.path.append(str(library_path))\n",
    "    # load_dotenv(str(library_path) + \"/credentials.env\")\n",
    "    # os.environ[\"AZURE_OPENAI_MODEL_NAME\"] = os.environ[\"GPT35_DEPLOYMENT_NAME\"]\n",
    "\n",
    "    ###################################\n",
    "```\n",
    "2) Open a terminal, activate the right conda environment, then go to this folder `apps/backend/langserve/app` and run this command:\n",
    "    \n",
    "```bash\n",
    "python server.py\n",
    "```\n",
    "\n",
    "Alternatively, you can go to this folder `apps/backend/langserve/` and run this command:\n",
    "```bash\n",
    "langchain serve\n",
    "```\n",
    "\n",
    "This will run the backend server API in localhost port 8000. \n",
    "\n",
    "3) If you are working on an Azure ML compute instance you can access the OpenAPI (Swagger) definition in this address:\n",
    "\n",
    "    https:\\<your_compute_name\\>-8000.\\<your_region\\>.instances.azureml.ms/\n",
    "    \n",
    "    for example:\n",
    "    https://pabmar1-8000.australiaeast.instances.azureml.ms/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879ecc10-6023-4157-8a7d-b8e5ec6e8509",
   "metadata": {},
   "source": [
    "## Talk to the API using POST requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8ab5b242-31dd-42f2-b24b-4489a77cd5a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import sys\n",
    "import time\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53dccb57-e718-42d0-804f-8aebbf44d8b2",
   "metadata": {},
   "source": [
    "### Functions to post and read responses from the API. It supports streaming!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fa825ba4-57b5-40ec-b530-09592c0c5971",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_line(line):\n",
    "    \"\"\"Process a single line from the stream.\"\"\"\n",
    "    # print(\"line:\",line)\n",
    "    if line.startswith('data: '):\n",
    "        # Extract JSON data following 'data: '\n",
    "        json_data = line[len('data: '):]\n",
    "        try:\n",
    "            data = json.loads(json_data)\n",
    "            if \"event\" in data:\n",
    "                handle_event(data)\n",
    "            elif \"content\" in data:\n",
    "                # If there is immediate content to print\n",
    "                print(data[\"content\"], end=\"\", flush=True)\n",
    "            elif \"steps\" in data:\n",
    "                print(data[\"steps\"])\n",
    "            elif \"output\" in data:\n",
    "                print(data[\"output\"])\n",
    "        except json.JSONDecodeError as e:\n",
    "            print(f\"JSON decoding error: {e}\")\n",
    "    elif line.startswith('event: '):\n",
    "        pass\n",
    "    elif \": ping\" in line:\n",
    "        pass\n",
    "    else:\n",
    "        print(line)\n",
    "\n",
    "def handle_event(event):\n",
    "    \"\"\"Handles specific events, adjusting output based on event type.\"\"\"\n",
    "    kind = event[\"event\"]\n",
    "    if kind == \"on_chain_start\" and event[\"name\"] == \"AgentExecutor\":\n",
    "        print(f\"Starting agent: {event['name']}\")\n",
    "    elif kind == \"on_chain_end\" and event[\"name\"] == \"AgentExecutor\":\n",
    "        print(\"\\n--\")\n",
    "        print(f\"Done agent: {event['name']}\")\n",
    "    elif kind == \"on_chat_model_stream\":\n",
    "        content = event[\"data\"][\"chunk\"][\"content\"]\n",
    "        if content:  # Ensure content is not None or empty\n",
    "            print(content, end=\"\", flush=True)\n",
    "    elif kind == \"on_tool_start\":\n",
    "        # Assuming event['data'].get('input') is a dictionary\n",
    "        tool_inputs = event['data'].get('input')\n",
    "        if isinstance(tool_inputs, dict):\n",
    "            # Joining the dictionary into a string format key: 'value'\n",
    "            inputs_str = \", \".join(f\"'{v}'\" for k, v in tool_inputs.items())\n",
    "        else:\n",
    "            # Fallback if it's not a dictionary or in an unexpected format\n",
    "            inputs_str = str(tool_inputs)\n",
    "        print(f\"Starting tool: {event['name']} with input: {inputs_str}\")\n",
    "    elif kind == \"on_tool_end\":\n",
    "        print(f\"Done tool: {event['name']}\\n--\")\n",
    "\n",
    "    \n",
    "def consume_api(url, payload):\n",
    "    \"\"\"Uses requests POST to talkt to the FastAPI backend, supports streaming\"\"\"\n",
    "    \n",
    "    headers = {'Content-Type': 'application/json'}\n",
    "    \n",
    "    with requests.post(url, json=payload, headers=headers, stream=True) as response:\n",
    "        try:\n",
    "            response.raise_for_status()  # Raises a HTTPError if the response is not 200\n",
    "            \n",
    "            for line in response.iter_lines():\n",
    "                if line:  # Check if the line is not empty\n",
    "                    decoded_line = line.decode('utf-8')\n",
    "                    process_line(decoded_line)\n",
    "                    \n",
    "                    \n",
    "        except requests.exceptions.HTTPError as err:\n",
    "            print(f\"HTTP Error: {err}\")\n",
    "        except Exception as e:\n",
    "            print(f\"An error occurred: {e}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6adc3d3-2a27-47de-8984-926db605e259",
   "metadata": {},
   "source": [
    "### Base URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "53845890-5010-4b92-bdb3-2f4635d92290",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_url = \"https://<YOUR_BACKEND_WEBAPP_NAME>-staging.azurewebsites.net\"  # Note that \"-staging\" is the Azure App Service slot where the LangServe API was deployed\n",
    "# base_url = \"http://localhost:8000\" # If you deployed locally"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d0baf2a-bdc4-46e2-9a86-ea964eaad98c",
   "metadata": {},
   "source": [
    "### `/chatgpt/` endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d466c935-5209-4759-8710-9d8333b0ac1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = {'input': 'explain long covid in just 2 short sentences'}  # Your POST request payload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "82a77ba7-6e28-48e0-b064-5a08518f03a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"output\":{\"content\":\"Long COVID refers to a range of symptoms that persist for weeks or months after the initial infection with COVID-19. These symptoms can include fatigue, shortness of breath, and cognitive difficulties.\",\"additional_kwargs\":{},\"response_metadata\":{\"token_usage\":{\"completion_tokens\":38,\"prompt_tokens\":16,\"total_tokens\":54},\"model_name\":\"gpt-35-turbo\",\"system_fingerprint\":\"fp_2f57f81c11\",\"prompt_filter_results\":[{\"prompt_index\":0,\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}],\"finish_reason\":\"stop\",\"logprobs\":null,\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}},\"type\":\"ai\",\"name\":null,\"id\":\"run-c6276c52-6d60-4c94-82bb-c0cc2c9eaf0b-0\",\"example\":false,\"tool_calls\":[],\"invalid_tool_calls\":[]},\"metadata\":{\"run_id\":\"c6276c52-6d60-4c94-82bb-c0cc2c9eaf0b\",\"feedback_tokens\":[]}}\n"
     ]
    }
   ],
   "source": [
    "# URL of the FastAPI Invoke endpoint\n",
    "url = base_url + '/chatgpt/invoke'\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0cad6f28-9397-440c-816c-6bea4ed4d417",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Long COVID is a condition where people continue to experience symptoms of COVID-19 long after the initial infection has cleared. These symptoms can include fatigue, shortness of breath, and brain fog."
     ]
    }
   ],
   "source": [
    "# URL of the FastAPI streaming endpoint\n",
    "url = base_url + '/chatgpt/stream'\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8324dbea-a7d4-408d-b1c0-b4852a8a3163",
   "metadata": {},
   "source": [
    "### `/joke` endpoint : chain with custom output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f018c89b-99cc-4ebe-9ae0-475f9fe818dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"output\":{\"content\":\"Why don't you ever hear a pterodactyl using the bathroom in high school? Because they're extinct!\",\"info\":{\"timestamp\":\"2024-04-19T22:21:24.182157\"}},\"metadata\":{\"run_id\":\"da46e2af-5363-4f65-b91c-80db8c82cea5\",\"feedback_tokens\":[]}}\n"
     ]
    }
   ],
   "source": [
    "payload = {'input': {\"topic\": \"highschool\", \"language\":\"english\"}}\n",
    "\n",
    "url = base_url + '/joke/invoke'\n",
    "\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d7973395-60ee-4c2a-8bee-51851bb55374",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Why did the math book go to high school?\n",
      "\n",
      "It wanted to become well-rounded!"
     ]
    }
   ],
   "source": [
    "# URL of the FastAPI streaming endpoint\n",
    "url = base_url + '/joke/stream_events'\n",
    "\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "040e150d-8837-43f4-ae50-cdc687097fd4",
   "metadata": {},
   "source": [
    "### `/agent` endpoint : our complex smart bot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2edd4a54-e64d-45d0-94d8-bd7d5b9ff660",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "session462 user68\n"
     ]
    }
   ],
   "source": [
    "random_session_id = \"session\"+ str(random.randint(1, 1000))\n",
    "ramdom_user_id = \"user\"+ str(random.randint(1, 1000))\n",
    "\n",
    "config={\"configurable\": {\"session_id\": random_session_id, \"user_id\": ramdom_user_id}}\n",
    "print(random_session_id, ramdom_user_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d893c0eb-a069-4502-b178-0da6e3341e78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"output\":{\"output\":\"I'm here to assist you with any questions or tasks you have. How can I help you today?\"},\"metadata\":{\"run_id\":\"fae317ea-2efd-4f85-b145-65ca4878df0a\",\"feedback_tokens\":[]}}\n"
     ]
    }
   ],
   "source": [
    "payload = {'input': {\"question\": \"Hi, I am Pablo, what is your name?\"}, 'config': config}\n",
    " \n",
    "url = base_url + '/agent/invoke'\n",
    "\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8e9fb7c2-1ffc-4b74-9c72-2d971813c666",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting agent: AgentExecutor\n",
      "Starting tool: docsearch with input: 'CLP'\n",
      "Done tool: docsearch\n",
      "--\n",
      "I have found multiple meanings and applications of the term \"CLP\" in different contexts. Here are the different meanings and their respective sources:\n",
      "\n",
      "1. **Constraint Logic Programming (CLP):**\n",
      "   - **Definition:** Constraint Logic Programming (CLP) is a powerful extension of conventional logic programming that incorporates constraint languages and constraint solving methods into logic programming languages.\n",
      "   - **Key Concepts:** CLP involves the parametrization of a logic programming language with respect to a constraint language and a domain of computation, yielding soundness and completeness results for an operational semantics relying on a constraint solver for the employed constraint language.\n",
      "   - **Source:** [arXiv:cs/0008036v1](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0008/0008036v1.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-01-03T02:11:44Z&st=2024-01-02T18:11:44Z&spr=https&sig=ngrEqvqBVaxyuSYqgPVeF%2B9c0fXLs94v3ASgwg7LDBs%3D)\n",
      "\n",
      "2. **Core-Like Particles (CLP):**\n",
      "   - **Definition:** CLP refers to core-like particles, specifically in the context of quantifying recombinant baculovirus-generated bluetongue virus (BTV) core-like particles in purified preparations or lysates of recombinant baculovirus-infected cells.\n",
      "   - **Measurement:** The CLP concentration in purified preparations was determined to be 6.6 x 10(15) particles/l, and in lysates of recombinant baculovirus-infected cells, it was determined to reach a value of 3 x 10(15) particles/l of culture medium at 96 h post-infection.\n",
      "   - **Source:** [PubMed](https://www.ncbi.nlm.nih.gov/pubmed/10403670/?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-01-03T02:11:44Z&st=2024-01-02T18:11:44Z&spr=https&sig=ngrEqvqBVaxyuSYqgPVeF%2B9c0fXLs94v3ASgwg7LDBs%3D)\n",
      "\n",
      "3. **Recurrence with Affine Level Mappings in CLP(R):**\n",
      "   - **Definition:** This context refers to a technical note discussing the decidability of termination for Constraint Logic Programming over the real numbers (CLP(R)) using affine level mappings.\n",
      "   - **Key Concepts:** The paper introduces a class of constraint logic programs such that their termination can be proved by using affine level mappings, and it shows that membership to this class is decidable in polynomial time.\n",
      "   - **Source:** [arXiv:cs/0701082v1](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0701/0701082v1.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-01-03T02:11:44Z&st=2024-01-02T18:11:44Z&spr=https&sig=ngrEqvqBVaxyuSYqgPVeF%2B9c0fXLs94v3ASgwg7LDBs%3D)\n",
      "\n",
      "Each of these contexts provides a distinct meaning and application of the term \"CLP.\" If you have a specific context in mind, please let me know so that I can provide more detailed information.\n",
      "--\n",
      "Done agent: AgentExecutor\n"
     ]
    }
   ],
   "source": [
    "payload = {'input': {\"question\": \"docsearch, what is CLP?\"}, 'config': config}\n",
    " \n",
    "url = base_url + '/agent/stream_events'\n",
    "\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c0d4555d-6c09-465f-9c02-c3c194b8588b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting agent: AgentExecutor\n",
      "Starting tool: bing with input: 'current salary of a dental hygienist in Texas'\n",
      "Done tool: bing\n",
      "--\n",
      "I have found various estimates for the average salary of a dental hygienist in Texas from different sources. Here are the estimates:\n",
      "\n",
      "1. **Indeed:** The average salary for a dental hygienist is $46.88 per hour in Texas, based on 2.8k reported salaries. [Source](https://www.indeed.com/career/dental-hygienist/salaries/TX)\n",
      "\n",
      "2. **Salary.com:** The average salary for a dental hygienist in Texas is $81,346 as of March 26, 2024, with a range typically falling between $71,983 and $91,085. [Source](https://www.salary.com/research/salary/benchmark/dental-hygienist-salary/tx)\n",
      "\n",
      "3. **Glassdoor:** The highest reported salary for a dental hygienist in Texas is $130,309 per year, based on anonymous submissions. [Source](https://www.glassdoor.com/Salaries/texas-dental-hygienist-salary-SRCH_IL.0,5_IS1347_KO6,22_IP4.htm)\n",
      "\n",
      "4. **ZipRecruiter:** The average hourly pay for a dental hygienist in Texas is $40.35, with salaries ranging from $34.04 to $44.81 per hour. [Source](https://www.ziprecruiter.com/Salaries/Dental-Hygienist-Salary--in-Texas)\n",
      "\n",
      "5. **CareerExplorer:** The average salary for dental hygienists in Texas is around $75,640 per year, with salaries typically starting from $55,790 and going up to $97,920. [Source](https://www.careerexplorer.com/careers/dental-hygienist/salary/texas/)\n",
      "\n",
      "6. **DentalPost:** The average salary for a dental hygienist in Texas is $79,690 ($38.31 per hour) as of 2022, with the range typically falling between $76,750 and $88,250. [Source](https://www.dentalpost.net/salary-survey/average-dental-hygienist-salary-for-texas/)\n",
      "\n",
      "Please note that these figures are estimates and can vary based on factors such as experience, location, and employer. For more detailed and specific information, it's advisable to consult with local dental hygienist associations or professional organizations in Texas.\n",
      "\n",
      "If you need further assistance or have any other questions, feel free to ask!\n",
      "--\n",
      "Done agent: AgentExecutor\n"
     ]
    }
   ],
   "source": [
    "payload = {'input': {\"question\": \"bing, give me the current salary of a dental hygenist in texas\"}, 'config': config}\n",
    " \n",
    "url = base_url + '/agent/stream_events'\n",
    "\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d00491de-a3f9-46e6-b221-696968c0c1bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting agent: AgentExecutor\n",
      "Starting tool: docsearch with input: 'How does COVID-19 affect obese people?'\n",
      "Starting tool: docsearch with input: 'How does COVID-19 affect elderly people?'\n",
      "Done tool: docsearch\n",
      "--\n",
      "Done tool: docsearch\n",
      "--\n",
      "### How COVID-19 Affects Obese People\n",
      "\n",
      "Obesity has been identified as a significant risk factor for the severity of COVID-19, leading to more serious symptoms and negative prognoses for infected individuals. Here are some key impacts of COVID-19 on obese individuals:\n",
      "\n",
      "1. **Increased Risk of Severe Disease**: Studies have shown that obese patients with COVID-19 have increased odds of progressing to severe disease. They are more likely to exhibit symptoms such as cough and fever compared to non-obese patients. Additionally, men who are obese have increased odds of developing severe COVID-19 compared to those with normal weight.\n",
      "\n",
      "2. **Higher Likelihood of Serious Complications**: The World Health Organization (WHO) considers obesity as a major risk factor for becoming seriously ill with COVID-19. Data from the UK Intensive Care National Audit and Research Centre indicated that two-thirds of people who developed serious or fatal COVID-19-related complications were overweight or obese.\n",
      "\n",
      "3. **Role of Obesity in Pathogenesis**: Obesity plays a significant role in the pathogenesis of COVID-19 patients, leading to consequences such as inflammation of adipose tissue, insulin resistance, and hypertension due to metabolic dysfunction. This highlights the need for increased vigilance, testing priority, and therapy for COVID-19 patients with higher BMI and other co-morbidities, especially in the elderly population.\n",
      "\n",
      "These findings underscore the importance of closely managing and providing prompt and aggressive treatment to obese patients infected with COVID-19.\n",
      "\n",
      "### How COVID-19 Affects Elderly People\n",
      "\n",
      "COVID-19 significantly affects elderly people, leading to a greater incidence and severity of the disease. Here are some key impacts of COVID-19 on the elderly:\n",
      "\n",
      "1. **Higher Risk of Severe Illness and Mortality**: Elderly individuals are at a higher risk of experiencing more serious and potentially fatal illness associated with COVID-19. Mortality data indicates a significant risk of mortality for people in their 60s, 70s, and over 80s. In Spain, a high percentage of all coronavirus hospitalizations correspond to those over 60 years of age, highlighting the severity of the impact on the elderly.\n",
      "\n",
      "2. **Social Isolation and Vulnerability**: The global recommendation for older populations includes social isolation, which involves staying at home and avoiding contact with other people, possibly for an extended period of time. This is particularly important for people over 70 years, and 50 years in some particularly vulnerable Indigenous populations.\n",
      "\n",
      "3. **Biopsychosocial Care and Personalized Decisions**: There is a trend towards biopsychosocial care of elderly people in all settings, adapting care and personalizing decisions on hospital admissions, palliative care, and other criteria, to years adjusted to quality of life.\n",
      "\n",
      "4. **Transmission Reduction Measures**: Measures to reduce the transmission of the virus through hygiene and social distancing are necessary, attending to the biopsychosocial health of the elderly isolated. Intersectoral communication and the use of technological tools, accompanied by adequate digital health literacy, are proposed as innovative alternatives to support the elderly population.\n",
      "\n",
      "5. **Impact on Drug Therapy and Clinical Therapeutics**: The COVID-19 pandemic presents medical and social issues for older people, impacting the safety and efficacy of drug therapy. This includes implications for pharmacy practice, clinical therapeutics, and possible new treatments for the virus.\n",
      "\n",
      "These findings underscore the need for targeted interventions and support for the elderly population to mitigate the impact of COVID-19 on this vulnerable demographic.\n",
      "\n",
      "References:\n",
      "1. [Association of Obesity with Severity of COVID-19](https://doi.org/10.2337/dc20-0576)\n",
      "2. [Impact of Obesity on Seriously Ill COVID-19 Patients](https://doi.org/10.1002/oby.22844)\n",
      "3. [Role of Obesity in the Pathogenesis of COVID-19](http://medrxiv.org/cgi/content/short/2020.05.11.20098806v1?rss=1)\n",
      "4. [Impact of COVID-19 on Elderly People](https://doi.org/10.1016/j.enfcli.2020.05.004)\n",
      "5. [Impact of COVID-19 on Drug Therapy and Clinical Therapeutics](https://doi.org/10.4140/tcp.n.2020.190)\n",
      "\n",
      "If you have any more questions or need further information, feel free to ask!\n",
      "--\n",
      "Done agent: AgentExecutor\n"
     ]
    }
   ],
   "source": [
    "payload = {'input': {\"question\": \"docsearch, How Covid affects obese people? and elderly\"}, 'config': config}\n",
    " \n",
    "url = base_url + '/agent/stream_events'\n",
    "\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "22e2a06a-9310-4830-95a6-6772f8478849",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting agent: AgentExecutor\n",
      "Starting tool: sqlsearch with input: 'number of people hospitalized in California'\n",
      "Done tool: sqlsearch\n",
      "--\n",
      "The total number of people currently hospitalized in California is 2,653,612.\n",
      "\n",
      "This information was obtained by querying the `covidtracking` table for the sum of the `hospitalizedCurrently` column where the state is 'CA'. The SQL query used for this purpose is:\n",
      "\n",
      "```sql\n",
      "SELECT SUM(hospitalizedCurrently) AS total_hospitalized FROM covidtracking WHERE state = 'CA'\n",
      "```\n",
      "\n",
      "If you need further information or assistance, feel free to ask!\n",
      "--\n",
      "Done agent: AgentExecutor\n"
     ]
    }
   ],
   "source": [
    "payload = {'input': {\"question\": \"sqlsearch, how many people were hospitalized in CA?\"}, 'config': config}\n",
    " \n",
    "url = base_url + '/agent/stream_events'\n",
    "\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e28d0220-b01c-4d41-bcb2-ad7763935a7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting agent: AgentExecutor\n",
      "You're welcome! If you have any more questions or if there's anything else I can help you with, feel free to ask.\n",
      "--\n",
      "Done agent: AgentExecutor\n"
     ]
    }
   ],
   "source": [
    "payload = {'input': {\"question\": \"thank you!\"}, 'config': config}\n",
    " \n",
    "url = base_url + '/agent/stream_events'\n",
    "\n",
    "consume_api(url, payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6139eafe-5470-4c36-abcc-48931f18a66a",
   "metadata": {},
   "source": [
    "## Now let's try all endpoints and routes using langchain local RemoteRunnable\n",
    "\n",
    "All these are also available in TypeScript, see LangServe.JS documentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "50de072f-13b8-4144-a988-08007d3a5079",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.schema import SystemMessage, HumanMessage\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langchain.schema.runnable import RunnableMap\n",
    "from langserve import RemoteRunnable\n",
    "\n",
    "chatgpt_chain = RemoteRunnable(base_url + \"/chatgpt/\")\n",
    "joke_chain = RemoteRunnable(base_url + \"/joke/\")\n",
    "agent_chain = RemoteRunnable(base_url + \"/agent/\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "5242d7ed-2bde-40ec-94f9-7800d3bdc326",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'content': 'Why did the car break up with the motorcycle?\\nBecause it was tired of being co-dependent!',\n",
       " 'info': {'timestamp': '2024-04-19T22:25:56.305034'}}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joke_chain.invoke({\"topic\": \"cars\", \"language\":\"english\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "619229ef-0675-486b-9a6f-ff1cc6ec465d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'content': '¿Por qué los loros no saben contar chistes? Porque siempre repiten los mismos.',\n",
       " 'info': {'timestamp': '2024-04-19T22:26:00.392746'}}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or async\n",
    "await joke_chain.ainvoke({\"topic\": \"parrots\", \"language\":\"spanish\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "1b8f5ac1-91ad-4dfe-91ca-37acc74ea222",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Long COVID, also known as post-acute sequelae of SARS-CoV-2 infection (PASC), refers to a range of symptoms that persist for weeks or months after the acute phase of a COVID-19 infection has resolved. These symptoms can include fatigue, shortness of breath, chest pain, joint pain, and brain fog, among others. The exact cause of long COVID is not yet fully understood, but it is believed to involve a combination of factors, including lingering viral effects, immune system dysregulation, and potential damage to organs or tissues. Long COVID can significantly impact a person's quality of life and may require ongoing medical care and support."
     ]
    }
   ],
   "source": [
    "prompt = [\n",
    "    SystemMessage(content='you are a helpful assistant that responds to the user question.'),\n",
    "    HumanMessage(content='explain long covid')\n",
    "]\n",
    "\n",
    "# Supports astream\n",
    "async for msg in chatgpt_chain.astream(prompt):\n",
    "    print(msg.content, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "bad4ed02-3a11-41fd-b1da-7823d87eaa60",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting agent: AgentExecutor\n",
      "--\n",
      "Starting tool: booksearch with inputs: {'query': 'stolen kidney'}\n",
      "Done tool: booksearch\n",
      "--\n",
      "The concept of a \"stolen kidney\" is often associated with an urban legend known as the \"Kidney Heist tale.\" This urban legend typically involves a scenario where an individual is drugged, wakes up in an ice-filled bathtub, and discovers that one of their kidneys has been surgically removed. The story is often used as a cautionary tale about accepting drinks from strangers and has circulated in various versions over the years, each sharing the core elements of the drugged drink, the ice-filled bathtub, and the kidney-theft punch line.\n",
      "\n",
      "The Kidney Heist tale is an example of a story that sticks in people's minds. It is memorable, understandable, and effective in changing thought or behavior. This urban legend shares many traits with other successful ideas, such as unexpected outcomes, concrete details, and emotional impact. The story's ability to stick in people's minds is attributed to its vivid and memorable nature, making it easy to understand, remember, and retell later.\n",
      "\n",
      "While the \"stolen kidney\" concept is often associated with this urban legend, it is important to note that the Kidney Heist tale is a fictional story and not based on real events. It serves as an example of how certain stories, whether true or not, can have a lasting impact on individuals and influence their behavior.\n",
      "\n",
      "The association of the \"stolen kidney\" concept with this urban legend highlights the power of storytelling and the factors that contribute to the memorability and impact of certain narratives.\n",
      "\n",
      "For further details and insights into the concept of sticky ideas and the Kidney Heist tale, you can refer to the source document: [Made To Stick](https://datasetsgptsmartsearch.blob.core.windows.net/books/Made_To_Stick.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-01-03T02:11:44Z&st=2024-01-02T18:11:44Z&spr=https://sig=ngrEqvqBVaxyuSYqgPVeF%2B9c0fXLs94v3ASgwg7LDBs%3D).\n",
      "\n",
      "If you have any more questions or need further information, feel free to ask!\n",
      "--\n",
      "Done agent: AgentExecutor\n"
     ]
    }
   ],
   "source": [
    "async for event in agent_chain.astream_events({\"question\": \" booksearch, what is the story about the stolen kidney, and what book is it in?\"}, config=config, version=\"v1\"):\n",
    "    kind = event[\"event\"]\n",
    "    if kind == \"on_chain_start\":\n",
    "        if (event[\"name\"] == \"AgentExecutor\"):  \n",
    "            print(f\"Starting agent: {event['name']}\")\n",
    "    elif kind == \"on_chain_end\":\n",
    "        if (event[\"name\"] == \"AgentExecutor\"):\n",
    "            print()\n",
    "            print(\"--\")\n",
    "            print(f\"Done agent: {event['name']}\")\n",
    "    if kind == \"on_chat_model_stream\":\n",
    "        content = event[\"data\"][\"chunk\"].content\n",
    "        if content:\n",
    "            print(content, end=\"\", flush=True)\n",
    "    elif kind == \"on_tool_start\":\n",
    "        print(\"--\")\n",
    "        print(f\"Starting tool: {event['name']} with inputs: {event['data'].get('input')}\")\n",
    "    elif kind == \"on_tool_end\":\n",
    "        print(f\"Done tool: {event['name']}\")\n",
    "        # print(f\"Tool output was: {event['data'].get('output')}\")\n",
    "        print(\"--\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6c538f1-333b-4c5d-a6a9-faebe75d381c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}