LlamaIndex_Mistral7B-RAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mistral 7B\n",
    "#### RAG with LlamaIndex - Nvidia CUDA + WSL (Windows Subsystem for Linux) + Word documents + Local LLM\n",
    "\n",
    "This notebook demonstrates the use of LlamaIndex for Retrieval Augmented Generation using Windows' WSL and an Nvidia's CUDA.\n",
    "\n",
    "See the [README.md](README.md) file for help on how to run this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Prepare Llama Index for use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/markwsl/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import sys\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
    "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
    "\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, ServiceContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Load the Word document(s)\n",
    "\n",
    "Note: A fictitious story about Thundertooth a dinosaur who has travelled to the future. Thanks ChatGPT!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader(\"./Data/\").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Instantiate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes\n",
      "llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from ./Models/mistral-7b-instruct-v0.1.Q6_K.gguf (version GGUF V2)\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = mistralai_mistral-7b-instruct-v0.1\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 32768\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000\n",
      "llama_model_loader: - kv  11:                          general.file_type u32              = 18\n",
      "llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q6_K:  226 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V2\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 32768\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_embd_head_k    = 128\n",
      "llm_load_print_meta: n_embd_head_v    = 128\n",
      "llm_load_print_meta: n_gqa            = 4\n",
      "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
      "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 14336\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 32768\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = Q6_K\n",
      "llm_load_print_meta: model params     = 7.24 B\n",
      "llm_load_print_meta: model size       = 5.53 GiB (6.56 BPW) \n",
      "llm_load_print_meta: general.name     = mistralai_mistral-7b-instruct-v0.1\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size =    0.22 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =   102.54 MiB\n",
      "llm_load_tensors:      CUDA0 buffer size =  5563.55 MiB\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 8192\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init:      CUDA0 KV buffer size =  1024.00 MiB\n",
      "llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB\n",
      "llama_new_context_with_model:  CUDA_Host input buffer size   =    25.07 MiB\n",
      "llama_new_context_with_model:      CUDA0 compute buffer size =   560.03 MiB\n",
      "llama_new_context_with_model:  CUDA_Host compute buffer size =     8.00 MiB\n",
      "llama_new_context_with_model: graph splits (measure): 3\n",
      "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | \n",
      "Model metadata: {'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '32768', 'general.name': 'mistralai_mistral-7b-instruct-v0.1', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '18'}\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "from llama_index.llms.llama_cpp import LlamaCPP\n",
    "from llama_index.llms.llama_cpp.llama_utils import messages_to_prompt, completion_to_prompt\n",
    "llm = LlamaCPP(\n",
    "    model_url=None, # We'll load locally.\n",
    "    model_path='./Models/mistral-7b-instruct-v0.1.Q6_K.gguf', # 6-bit model\n",
    "    temperature=0.1,\n",
    "    max_new_tokens=1024, # Increasing to support longer responses\n",
    "    context_window=8192, # Mistral7B has an 8K context-window\n",
    "    generate_kwargs={},\n",
    "    # set to at least 1 to use GPU\n",
    "    model_kwargs={\"n_gpu_layers\": 40}, # 40 was a good amount of layers for the RTX 3090, you may need to decrease yours if you have less VRAM than 24GB\n",
    "    messages_to_prompt=messages_to_prompt,\n",
    "    completion_to_prompt=completion_to_prompt,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Checkpoint\n",
    "\n",
    "Are you running on GPU? The above output should include near the top something like:\n",
    "> ggml_init_cublas: found 1 CUDA devices:\n",
    "\n",
    "And in the full text near the bottom should be:\n",
    "> llm_load_tensors: using CUDA for GPU acceleration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Embeddings\n",
    "\n",
    "Convert your source document text into embeddings.\n",
    "\n",
    "The embedding model is from huggingface, this one performs well.\n",
    "\n",
    "> https://huggingface.co/thenlper/gte-large\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "\n",
    "embed_model = HuggingFaceEmbedding(model_name=\"thenlper/gte-large\", cache_folder=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Prompt Template\n",
    "\n",
    "Prompt template for Mistral:\n",
    "\n",
    ">     <s>[INST] {prompt} [/INST]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Service Context\n",
    "\n",
    "For chunking the document into tokens using the embedding model and our LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "Settings.chunk_size=256 # Number of tokens in each chunk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Index documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = VectorStoreIndex.from_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. Query Engine\n",
    "\n",
    "Create a query engine, specifying how many citations we want to get back from the searched text (in this case 3).\n",
    "\n",
    "The DB_DOC_ID_KEY is used to get back the filename of the original document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine import CitationQueryEngine\n",
    "query_engine = CitationQueryEngine.from_args(\n",
    "    index,\n",
    "    similarity_top_k=3,\n",
    "    # here we can control how granular citation sources are, the default is 512\n",
    "    citation_chunk_size=256,\n",
    ")\n",
    "\n",
    "# For citations we get the document info\n",
    "DB_DOC_ID_KEY = \"db_document_id\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. Prompt and Response function\n",
    "\n",
    "Pass in a question, get a response back.\n",
    "\n",
    "IMPORTANT: The prompt is in the question, adjust it to match what you want the LLM to act like and do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def RunQuestion(questionText):\n",
    "    queryQuestion = \"<s>[INST] You are a technology specialist. Answer questions in a positive, helpful and empathetic way. Answer the following question: \" + questionText + \" [/INST]\"\n",
    "\n",
    "    response = query_engine.query(queryQuestion)\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 11. Questions to test with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "TestQuestions = [\n",
    "    \"Summarise the story for me\",\n",
    "    \"Who was the main protagonist?\",\n",
    "    \"Did they have any children? If so, what were their names?\",\n",
    "    \"Did anything eventful happen?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12. Run Questions through model (this can take a while) and see citations\n",
    "\n",
    "Runs each test question, saves it to a dictionary for output in the last step.\n",
    "\n",
    "Note: Citations are the source documents used and the text the response is based on. This is important for RAG so you can reference these documents for the user, and to ensure it's utilising the right documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1/4: Summarise the story for me\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     164.72 ms\n",
      "llama_print_timings:      sample time =      50.72 ms /   191 runs   (    0.27 ms per token,  3766.00 tokens per second)\n",
      "llama_print_timings: prompt eval time =     375.64 ms /  1084 tokens (    0.35 ms per token,  2885.71 tokens per second)\n",
      "llama_print_timings:        eval time =    2749.26 ms /   190 runs   (   14.47 ms per token,    69.11 tokens per second)\n",
      "llama_print_timings:       total time =    3551.24 ms /  1274 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/3: |Thundertooth Part 2.docx| Source 1:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "2/3: |Thundertooth Part 1.docx| Source 2:\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "\n",
      "\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Grateful for the mayor's hospitality, Thundertooth followed her through the city. Together, they explored the futuristic marketplaces and innovative food labs, eventually discovering a sustainable solution that satisfied the dinosaur's hunger without compromising the well-being of the city's inhabitants.\n",
      "\n",
      "3/3: |Thundertooth Part 2.docx| Source 3:\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "2/4: Who was the main protagonist?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     164.72 ms\n",
      "llama_print_timings:      sample time =       3.72 ms /    14 runs   (    0.27 ms per token,  3765.47 tokens per second)\n",
      "llama_print_timings: prompt eval time =     214.88 ms /   566 tokens (    0.38 ms per token,  2634.02 tokens per second)\n",
      "llama_print_timings:        eval time =     186.46 ms /    13 runs   (   14.34 ms per token,    69.72 tokens per second)\n",
      "llama_print_timings:       total time =     433.60 ms /   579 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/3: |Thundertooth Part 2.docx| Source 1:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "2/3: |Thundertooth Part 2.docx| Source 2:\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "3/3: |Thundertooth Part 2.docx| Source 3:\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "\n",
      "Echo: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\n",
      "\n",
      "\n",
      "3/4: Did they have any children? If so, what were their names?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     164.72 ms\n",
      "llama_print_timings:      sample time =      31.32 ms /   115 runs   (    0.27 ms per token,  3672.24 tokens per second)\n",
      "llama_print_timings: prompt eval time =     264.52 ms /   843 tokens (    0.31 ms per token,  3186.92 tokens per second)\n",
      "llama_print_timings:        eval time =    1648.03 ms /   114 runs   (   14.46 ms per token,    69.17 tokens per second)\n",
      "llama_print_timings:       total time =    2167.21 ms /   957 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/3: |Thundertooth Part 2.docx| Source 1:\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "2/3: |Thundertooth Part 2.docx| Source 2:\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "\n",
      "Echo: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\n",
      "\n",
      "3/3: |Thundertooth Part 2.docx| Source 3:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "4/4: Did anything eventful happen?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     164.72 ms\n",
      "llama_print_timings:      sample time =      34.71 ms /   128 runs   (    0.27 ms per token,  3687.38 tokens per second)\n",
      "llama_print_timings: prompt eval time =     270.67 ms /   862 tokens (    0.31 ms per token,  3184.74 tokens per second)\n",
      "llama_print_timings:        eval time =    1795.24 ms /   127 runs   (   14.14 ms per token,    70.74 tokens per second)\n",
      "llama_print_timings:       total time =    2348.92 ms /   989 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/3: |Thundertooth Part 2.docx| Source 1:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "2/3: |Thundertooth Part 3.docx| Source 2:\n",
      "As Ignis's controlled bursts of flames interacted with the meteor, it began to change course. The combined efforts of the Thundertooth family, guided by their unique talents, diverted the catastrophic collision. The meteor, once destined for destruction, now harmlessly sailed past the Earth, leaving the city and its inhabitants unscathed.\n",
      "\n",
      "\n",
      "\n",
      "The citizens, emerging from their shelters, erupted into cheers of gratitude. Mayor Grace approached Thundertooth, expressing her heartfelt thanks for the family's heroic efforts. The Thundertooth family, tired but triumphant, basked in the relief of having saved their beloved city from imminent disaster.\n",
      "\n",
      "\n",
      "\n",
      "In the wake of the crisis, the citizens of the futuristic city hailed Thundertooth and his family as true heroes. The toy factory that once brought joy to children now became a symbol of resilience and unity. The Thundertooth family's legacy was forever etched in the city's history, a testament to the power of cooperation and the extraordinary capabilities that could emerge when dinosaurs and humans worked together for the greater good.\n",
      "\n",
      "3/3: |Thundertooth Part 2.docx| Source 3:\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "qa_pairs = []\n",
    "\n",
    "for index, question in enumerate(TestQuestions, start=1):\n",
    "    question = question.strip() # Clean up\n",
    "\n",
    "    print(f\"\\n{index}/{len(TestQuestions)}: {question}\")\n",
    "\n",
    "    response = RunQuestion(question) # Query and get  response\n",
    "\n",
    "    qa_pairs.append((question.strip(), str(response).strip())) # Add to our output array\n",
    "\n",
    "    # Displays the citations\n",
    "    for index, node in enumerate(response.source_nodes, start=1):\n",
    "        print(f\"{index}/{len(response.source_nodes)}: |{node.node.metadata['file_name']}| {node.node.get_text()}\")\n",
    "\n",
    "    # Uncomment the following line if you want to test just the first question\n",
    "    # break "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 13. Output responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4 Summarise the story for me\n",
      "\n",
      "Thundertooth is a prehistoric dinosaur who finds himself in a futuristic city where he meets Mayor Grace. Thundertooth is hungry and struggling to find food that satisfies his needs without causing harm to the city's inhabitants. Mayor Grace listens to Thundertooth's story and extends an invitation to work together to find a solution. Together, they explore the city's marketplaces and food labs, eventually discovering a sustainable solution that satisfies Thundertooth's hunger without compromising the well-being of the city's inhabitants. Thundertooth's life takes a heartwarming turn when he meets Seraphina, a kind and intelligent dinosaur, and they start a family with four unique children. Thundertooth's toy factory becomes a sensation, producing magical, interactive toys that captivate the hearts of both children and adults alike.\n",
      "\n",
      "--------\n",
      "\n",
      "2/4 Who was the main protagonist?\n",
      "\n",
      "The main protagonist in the story is Thundertooth.\n",
      "\n",
      "--------\n",
      "\n",
      "3/4 Did they have any children? If so, what were their names?\n",
      "\n",
      "Yes, Thundertooth had four children with Seraphina. Their names were Lumina, Echo, Nova, and Rhythm. Lumina inherited her mother's intelligence and her father's sense of wonder. She had the ability to generate light at will and was fascinated with technology. Echo had a gift for mimicry and could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. Nova had the ability to manipulate time and space, while Rhythm had the ability to control sound waves.\n",
      "\n",
      "--------\n",
      "\n",
      "4/4 Did anything eventful happen?\n",
      "\n",
      "Based on the provided sources, it appears that there were several eventful happenings in the story of Thundertooth and his family. Thundertooth founded a successful toy factory that produced innovative widgets with advanced technology, which brought joy to children across the city. Later, the family saved their city from a meteor collision through their unique talents and cooperation with the citizens. The citizens hailed them as heroes, and their toy factory became a symbol of resilience and unity. Overall, the story highlights the power of creativity, intelligence, and cooperation in overcoming challenges and making a positive impact on the world.\n",
      "\n",
      "--------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, (question, answer) in enumerate(qa_pairs, start=1):\n",
    "    print(f\"{index}/{len(qa_pairs)} {question}\\n\\n{answer}\\n\\n--------\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llamaindexgeneric",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}