01-LangChain-RAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LangChain local LLM RAG example\n",
    "Utilising LangChain v0.1\n",
    "\n",
    "This notebook demonstrates the use of LangChain for Retrieval Augmented Generation in Linux with Nvidia's CUDA. LLMs are run using Ollama.\n",
    "\n",
    "Models tested:\n",
    "- Llama 2\n",
    "- Mistral 7B\n",
    "- Mixtral 8x7B\n",
    "- Neural Chat 7B\n",
    "- Orca 2\n",
    "- Phi-2\n",
    "- Solar 10.7B\n",
    "- Yi 34B\n",
    "\n",
    "\n",
    "See the [README.md](README.md) file for help on how to setup your environment to run this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select your model here, put the name of the model in the ollama_model_name variable\n",
    "# Ensure you have pulled them or run them so Ollama has downloaded them and can load them (which it will do automatically)\n",
    "\n",
    "# Ollama installation (if you haven't done it yet): $ curl https://ollama.ai/install.sh | sh\n",
    "# Models need to be running in Ollama for LangChain to use them, to test if it can be run: $ ollama run mistral:7b-instruct-q6_K\n",
    "\n",
    "ollama_model_name = \"orca2:13b-q5_K_S\"\n",
    "# \"llama2:7b-chat-q6_K\"\n",
    "# \"mistral:7b-instruct-q6_K\"\n",
    "# \"mixtral:8x7b-instruct-v0.1-q4_K_M\"\n",
    "# \"neural-chat:7b-v3.3-q6_K\"\n",
    "# \"orca2:13b-q5_K_S\"\n",
    "# \"phi\" or try \"phi:chat\"\n",
    "# \"solar:10.7b-instruct-v1-q5_K_M\"\n",
    "# Can't run \"yi:34b-chat-q3_K_M\" or \"yi:34b-chat-q4_K_M\" - never stopped with inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the LLM with Ollama, setting the temperature low so it's not too creative\n",
    "\n",
    "from langchain_community.llms import Ollama\n",
    "llm = Ollama(model=ollama_model_name, temperature=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The sky appears blue because of a phenomenon called Rayleigh scattering. When sunlight enters the atmosphere, it encounters molecules of nitrogen and oxygen. These molecules are much smaller than the wavelengths of visible light, so they can interact with different colors of light in different ways. Shorter wavelengths, such as violet and blue, are scattered more strongly than longer wavelengths, such as red and yellow. This means that more blue light reaches our eyes from all directions, making the sky look blue to us. However, at sunrise and sunset, the sun is lower in the sky, so we see more of the longer wavelengths that are scattered less by the atmosphere. This is why the sky looks redder at those times.'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Quick test of the LLM with a general question before we start doing RAG\n",
    "llm.invoke(\"why is the sky blue?\")\n",
    "\n",
    "# Note: This line would not complete for Yi-34B - need to work out why inferencing never finishes (works fine when running with the same prompt in ollama.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Embeddings will be based on the Ollama loaded model\n",
    "\n",
    "from langchain_community.embeddings import OllamaEmbeddings\n",
    "\n",
    "embeddings = OllamaEmbeddings(model=ollama_model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import DirectoryLoader\n",
    "\n",
    "loader = DirectoryLoader('Data', glob=\"**/*.docx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load documents\n",
    "\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Ensure we have the right number of Word documents loaded\n",
    "\n",
    "len(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split them up into chunks using a Text Splitter\n",
    "\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter()\n",
    "documents = text_splitter.split_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the embeddings from the chunks\n",
    "\n",
    "from langchain_community.vectorstores import FAISS\n",
    "\n",
    "vector = FAISS.from_documents(documents, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the prompt and then the chain\n",
    "\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "if ollama_model_name == \"phi\" or ollama_model_name == \"phi:chat\":\n",
    "    # Phi-2 prompt is less flexible\n",
    "    prompt_template = \"\"\"Instruct: With this context\\n\\n{context}\\n\\nQuestion: {input}\\nOutput:\"\"\"\n",
    "\n",
    "else:\n",
    "    prompt_template = \"\"\"You are a story teller, answering questions in an excited, insightful, and empathetic way. Answer the question based only on the provided context:\n",
    "\n",
    "    <context>\n",
    "    {context}\n",
    "    </context>\n",
    "\n",
    "    Question: {input}\"\"\"\n",
    "\n",
    "prompt = ChatPromptTemplate.from_template(prompt_template)\n",
    "document_chain = create_stuff_documents_chain(llm, prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RunnableBinding(bound=RunnableBinding(bound=RunnableAssign(mapper={\n",
       "  context: RunnableLambda(format_docs)\n",
       "}), config={'run_name': 'format_inputs'})\n",
       "| ChatPromptTemplate(input_variables=['context', 'input'], messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'input'], template='You are a story teller, answering questions in an excited, insightful, and empathetic way. Answer the question based only on the provided context:\\n\\n    <context>\\n    {context}\\n    </context>\\n\\n    Question: {input}'))])\n",
       "| Ollama(model='orca2:13b-q5_K_S', temperature=0.1)\n",
       "| StrOutputParser(), config={'run_name': 'stuff_documents_chain'})"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The LangChain chain\n",
    "document_chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the retriever and LangChain retriever chain\n",
    "\n",
    "from langchain.chains import create_retrieval_chain\n",
    "\n",
    "retriever = vector.as_retriever()\n",
    "retrieval_chain = create_retrieval_chain(retriever, document_chain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RunnableBinding(bound=RunnableAssign(mapper={\n",
       "  context: RunnableBinding(bound=RunnableLambda(lambda x: x['input'])\n",
       "           | VectorStoreRetriever(tags=['FAISS', 'OllamaEmbeddings'], vectorstore=<langchain_community.vectorstores.faiss.FAISS object at 0x7ff5d77ac7c0>), config={'run_name': 'retrieve_documents'})\n",
       "})\n",
       "| RunnableAssign(mapper={\n",
       "    answer: RunnableBinding(bound=RunnableBinding(bound=RunnableAssign(mapper={\n",
       "              context: RunnableLambda(format_docs)\n",
       "            }), config={'run_name': 'format_inputs'})\n",
       "            | ChatPromptTemplate(input_variables=['context', 'input'], messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'input'], template='You are a story teller, answering questions in an excited, insightful, and empathetic way. Answer the question based only on the provided context:\\n\\n    <context>\\n    {context}\\n    </context>\\n\\n    Question: {input}'))])\n",
       "            | Ollama(model='orca2:13b-q5_K_S', temperature=0.1)\n",
       "            | StrOutputParser(), config={'run_name': 'stuff_documents_chain'})\n",
       "  }), config={'run_name': 'retrieval_chain'})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Chain now incorporates the retriever\n",
    "retrieval_chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here are our test questions\n",
    "\n",
    "TestQuestions = [\n",
    "    \"Summarise the story for me\",\n",
    "    \"Who was the main protagonist?\",\n",
    "    \"Did they have any children? If so, what were their names?\",\n",
    "    \"Did anything eventful happen?\",\n",
    "    \"Who are the main characters?\",\n",
    "    \"What do you think happens next in the story?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want to see what's happening under the hood, set debug to True\n",
    "\n",
    "from langchain.globals import set_debug\n",
    "\n",
    "# set_debug(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1/6: Summarise the story for me\n",
      "\n",
      "2/6: Who was the main protagonist?\n",
      "\n",
      "3/6: Did they have any children? If so, what were their names?\n",
      "\n",
      "4/6: Did anything eventful happen?\n",
      "\n",
      "5/6: Who are the main characters?\n",
      "\n",
      "6/6: What do you think happens next in the story?\n"
     ]
    }
   ],
   "source": [
    "qa_pairs = []\n",
    "\n",
    "for index, question in enumerate(TestQuestions, start=1):\n",
    "    question = question.strip() # Clean up\n",
    "\n",
    "    print(f\"\\n{index}/{len(TestQuestions)}: {question}\")\n",
    "\n",
    "    response = retrieval_chain.invoke({\"input\": question})\n",
    "\n",
    "    qa_pairs.append((question.strip(), response[\"answer\"])) # Add to our output array\n",
    "\n",
    "    # Uncomment the following line if you want to test just the first question\n",
    "    # break "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/6 Summarise the story for me\n",
      "\n",
      "Possible summary:\n",
      "\n",
      "The story is about Thundertooth, a talking dinosaur who travels through time and ends up in a futuristic city where he finds a new home and starts a toy factory with the help of humans. He also meets Seraphina, his soulmate, and they have four children who inherit different abilities from both parents. The family faces challenges and adventures, such as finding food, saving the city from a meteor threat, and bringing joy to the people. The story shows how Thundertooth and his family bridge the gap between the past and the future, and how they use their talents for good.\n",
      "\n",
      "--------\n",
      "\n",
      "2/6 Who was the main protagonist?\n",
      "\n",
      "To answer the question, we need to identify the main character of the story, who is usually the one who faces the most challenges, changes, or conflicts. We can use clues from the context, such as the title, the introduction, and the conclusion, to find out who that is.\n",
      "\n",
      "The title of the story is \"Thundertooth\", which suggests that the main protagonist is a character named Thundertooth. The introduction confirms this by describing how Thundertooth was a talking dinosaur who traveled through time and ended up in a futuristic city, where he faced hunger, internal strife, and external threats. The conclusion also shows how Thundertooth became a beloved figure and a hero who saved the city from a meteor. Therefore, we can conclude that Thundertooth is the main protagonist of the story.\n",
      "\n",
      "### Final answer: Thundertooth\n",
      "\n",
      "--------\n",
      "\n",
      "3/6 Did they have any children? If so, what were their names?\n",
      "\n",
      "To answer this question based on the context, we need to follow these steps:\n",
      "\n",
      "- Locate the part of the context that mentions Thundertooth's family and children. This is in the paragraph that starts with \"Thundertooth found a one-of-a-kind toy factory...\" and ends with \"...the city's inhabitants, reminding them that sometimes, the most magical things could emerge from the most unexpected places.\"\n",
      "- Identify the names of Thundertooth's children. They are listed in the paragraph after the sentence \"As the years passed, Thundertooth's life took a heartwarming turn.\" The names are Lumina, Echo, Sapphire, and Ignis.\n",
      "- Confirm that they had any children by checking if the context mentions their birth or adoption. It does not explicitly state how they got their children, but it implies that they were born to Thundertooth and Seraphina, since they are described as \"their children\" in the next paragraph.\n",
      "- Summarize the answer in a brief sentence, using the information from the context.\n",
      "\n",
      "### Final answer: Yes, they had four children named Lumina, Echo, Sapphire, and Ignis.\n",
      "\n",
      "--------\n",
      "\n",
      "4/6 Did anything eventful happen?\n",
      "\n",
      "To answer this question, we need to review the context and identify any significant events or changes that occurred in the story of Thundertooth and his family. We can use the following steps:\n",
      "\n",
      "- Step 1: Scan the context for keywords or phrases that indicate an eventful occurrence, such as \"crisis\", \"disaster\", \"threat\", \"challenge\", \"adventure\", \"transformation\", etc.\n",
      "- Step 2: Evaluate each keyword or phrase and determine if it describes an eventful happening that affected the characters or the plot of the story. For example, a crisis could be eventful, but a false alarm might not be. A threat could be eventful, but a potential danger might not be.\n",
      "- Step 3: Summarize the main events or happenings that were eventful in the context, and explain why they were significant for the story. For example, we could say that Thundertooth's journey through time was an eventful happening that changed his life and led him to the futuristic city, where he faced a hunger dilemma and a meteor threat.\n",
      "- Step 4: Write a brief final answer that addresses the question based on the summary of the main events or happenings. For example, we could say: Yes, several eventful happenings occurred in the context, such as Thundertooth's time travel, his hunger dilemma, and the meteor threat. These happenings shaped the story and showed how Thundertooth and his family overcame challenges and brought unity to the city.\n",
      "\n",
      "### Final answer: Yes, several eventful happenings occurred in the context.\n",
      "\n",
      "--------\n",
      "\n",
      "5/6 Who are the main characters?\n",
      "\n",
      "One possible way to answer the question is:\n",
      "\n",
      "To answer the question, we need to identify the main characters in the story. The main characters are usually the ones who have a significant role in the plot, have distinctive personalities and traits, and face challenges or conflicts that shape their development. We can use these criteria to find the main characters in the context.\n",
      "\n",
      "The first paragraph introduces the Thundertooth family as the protagonists of the story. They are a group of dinosaurs who survived an internal strife and became a united and prosperous family. The paragraph mentions their names: Thundertooth, Seraphina, Lumina, Echo, Sapphire, and Ignis. These are the main characters of the story.\n",
      "\n",
      "The rest of the context follows the lives and adventures of the Thundertooth family in the futuristic city. It describes how they overcame their hunger dilemma, founded a toy factory, met new friends, and faced a meteor crisis. Throughout these events, the main characters show their unique abilities, talents, and personalities. They also learn from their experiences and grow as individuals and as a family.\n",
      "\n",
      "Therefore, based on the context, we can conclude that the main characters are Thundertooth, Seraphina, Lumina, Echo, Sapphire, and Ignis. ### Final answer: The main characters are Thundertooth, Seraphina, Lumina, Echo, Sapphire, and Ignis.\n",
      "\n",
      "--------\n",
      "\n",
      "6/6 What do you think happens next in the story?\n",
      "\n",
      "Possible answer:\n",
      "\n",
      "To answer this question, we need to use our imagination and creativity, as well as some clues from the context. We can think of different scenarios that could happen next in the story, such as:\n",
      "\n",
      "- The Thundertooth family receives recognition and rewards for their heroic deeds, such as medals, honors, or special privileges.\n",
      "- The Thundertooth family continues to innovate and create new toys, attracting more customers and fans from around the world.\n",
      "- The Thundertooth family faces new challenges and adventures, such as exploring other dimensions, meeting other talking dinosaurs, or encountering new enemies.\n",
      "- The Thundertooth family enjoys their peaceful and happy life, spending time with each other and their friends, and celebrating their achievements.\n",
      "\n",
      "There is no definitive answer to this question, as the story could go in many different directions. However, based on the tone and theme of the context, we can make some educated guesses about what might happen next. For example, since the context emphasizes the redemption and bonding of the Thundertooth family, we can assume that they will continue to work together and support each other in their future endeavors. Since the context also highlights the diversity and creativity of the family members, we can expect that they will keep inventing new things and having fun. And since the context shows the appreciation and admiration of the city's inhabitants for the Thundertooth family, we can imagine that they will receive some form of recognition and gratitude for their heroism.\n",
      "\n",
      "Therefore, a possible answer to the question is:\n",
      "\n",
      "### Final answer: The Thundertooth family receives recognition and rewards for their heroic deeds, such as medals, honors, or special privileges. They also continue to innovate and create new toys, attracting more customers and fans from around the world. They enjoy their peaceful and happy life, spending time with each other and their friends, and celebrating their achievements.\n",
      "\n",
      "--------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print out the questions and answers\n",
    "\n",
    "for index, (question, answer) in enumerate(qa_pairs, start=1):\n",
    "    print(f\"{index}/{len(qa_pairs)} {question}\\n\\n{answer}\\n\\n--------\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LangChainRAGLinux",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}