LlamaIndex_Solar-10.7B-Instruct-RAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solar 10.7B Instruct\n",
    "#### RAG with LlamaIndex - Nvidia CUDA + WSL (Windows Subsystem for Linux) + Word documents + Local LLM\n",
    "\n",
    "This notebook demonstrates the use of LlamaIndex for Retrieval Augmented Generation using Windows' WSL and an Nvidia's CUDA.\n",
    "\n",
    "See the [README.md](README.md) file for help on how to run this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Prepare Llama Index for use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/markwsl/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import sys\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
    "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
    "\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, ServiceContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Load the Word document(s)\n",
    "\n",
    "Note: A fictitious story about Thundertooth a dinosaur who has travelled to the future. Thanks ChatGPT!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader(\"./Data/\").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Instantiate the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes\n",
      "llama_model_loader: loaded meta data with 24 key-value pairs and 435 tensors from ./Models/solar-10.7b-instruct-v1.0.Q5_K_M.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 48\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000\n",
      "llama_model_loader: - kv  11:                          general.file_type u32              = 17\n",
      "llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32              = 2\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  22:                    tokenizer.chat_template str              = {% for message in messages %}{% if me...\n",
      "llama_model_loader: - kv  23:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   97 tensors\n",
      "llama_model_loader: - type q5_K:  289 tensors\n",
      "llama_model_loader: - type q6_K:   49 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_layer          = 48\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_embd_head_k    = 128\n",
      "llm_load_print_meta: n_embd_head_v    = 128\n",
      "llm_load_print_meta: n_gqa            = 4\n",
      "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
      "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 14336\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 34B\n",
      "llm_load_print_meta: model ftype      = Q5_K - Medium\n",
      "llm_load_print_meta: model params     = 10.73 B\n",
      "llm_load_print_meta: model size       = 7.08 GiB (5.66 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: PAD token        = 2 '</s>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size =    0.33 MiB\n",
      "llm_load_tensors: offloading 35 repeating layers to GPU\n",
      "llm_load_tensors: offloaded 35/49 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =  7245.24 MiB\n",
      "llm_load_tensors:      CUDA0 buffer size =  5133.59 MiB\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 4096\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init:  CUDA_Host KV buffer size =   208.00 MiB\n",
      "llama_kv_cache_init:      CUDA0 KV buffer size =   560.00 MiB\n",
      "llama_new_context_with_model: KV self size  =  768.00 MiB, K (f16):  384.00 MiB, V (f16):  384.00 MiB\n",
      "llama_new_context_with_model:  CUDA_Host input buffer size   =    17.04 MiB\n",
      "llama_new_context_with_model:      CUDA0 compute buffer size =   296.02 MiB\n",
      "llama_new_context_with_model:  CUDA_Host compute buffer size =   296.00 MiB\n",
      "llama_new_context_with_model: graph splits (measure): 5\n",
      "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | \n",
      "Model metadata: {'tokenizer.chat_template': \"{% for message in messages %}{% if message['role'] == 'system' %}{% if message['content']%}{{'### System:\\n' + message['content']+'\\n\\n'}}{% endif %}{% elif message['role'] == 'user' %}{{'### User:\\n' + message['content']+'\\n\\n'}}{% elif message['role'] == 'assistant' %}{{'### Assistant:\\n'  + message['content']}}{% endif %}{% if loop.last and add_generation_prompt %}{{ '### Assistant:\\n' }}{% endif %}{% endfor %}\", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.padding_token_id': '2', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '4096', 'general.name': 'LLaMA v2', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '48', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '17'}\n",
      "Using chat template: {% for message in messages %}{% if message['role'] == 'system' %}{% if message['content']%}{{'### System:\n",
      "' + message['content']+'\n",
      "\n",
      "'}}{% endif %}{% elif message['role'] == 'user' %}{{'### User:\n",
      "' + message['content']+'\n",
      "\n",
      "'}}{% elif message['role'] == 'assistant' %}{{'### Assistant:\n",
      "'  + message['content']}}{% endif %}{% if loop.last and add_generation_prompt %}{{ '### Assistant:\n",
      "' }}{% endif %}{% endfor %}\n",
      "Using chat eos_token: \n",
      "Using chat bos_token: \n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "from llama_index.llms.llama_cpp import LlamaCPP\n",
    "from llama_index.llms.llama_cpp.llama_utils import messages_to_prompt, completion_to_prompt\n",
    "llm = LlamaCPP(\n",
    "    model_url=None, # We'll load locally.\n",
    "    model_path='./Models/solar-10.7b-instruct-v1.0.Q5_K_M.gguf',\n",
    "    temperature=0.1,\n",
    "    max_new_tokens=1024, # Increasing to support longer responses\n",
    "    context_window=4096, # Context Window\n",
    "    generate_kwargs={},\n",
    "    # set to at least 1 to use GPU\n",
    "    model_kwargs={\"n_gpu_layers\": 35}, # 43 was a good amount of layers for the RTX 3090, you may need to decrease yours if you have less VRAM than 24GB\n",
    "    messages_to_prompt=messages_to_prompt,\n",
    "    completion_to_prompt=completion_to_prompt,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Checkpoint\n",
    "\n",
    "Are you running on GPU? The above output should include near the top something like:\n",
    "> ggml_init_cublas: found 1 CUDA devices:\n",
    "\n",
    "And in the full text near the bottom should be:\n",
    "> llm_load_tensors: using CUDA for GPU acceleration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Embeddings\n",
    "\n",
    "Convert your source document text into embeddings.\n",
    "\n",
    "The embedding model is from huggingface, this one performs well.\n",
    "\n",
    "> https://huggingface.co/thenlper/gte-large\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "\n",
    "embed_model = HuggingFaceEmbedding(model_name=\"thenlper/gte-large\", cache_folder=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Prompt Template\n",
    "\n",
    "Prompt template for SOLAR 10.7B Instruct:\n",
    "\n",
    "&#35;&#35;&#35; User:<br>\n",
    "{prompt}<br>\n",
    "\n",
    "&#35;&#35;&#35; Assistant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Produces a prompt specific to the model\n",
    "def modelspecific_prompt(promptmessage, questionText):\n",
    "    return f\"### User:\\n{promptmessage}{questionText}\\n\\n###Assistant:\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Service Context\n",
    "\n",
    "For chunking the document into tokens using the embedding model and our LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "Settings.chunk_size=256 # Number of tokens in each chunk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Index documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = VectorStoreIndex.from_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. Query Engine\n",
    "\n",
    "Create a query engine, specifying how many citations we want to get back from the searched text (in this case 3).\n",
    "\n",
    "The DB_DOC_ID_KEY is used to get back the filename of the original document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine import CitationQueryEngine\n",
    "query_engine = CitationQueryEngine.from_args(\n",
    "    index,\n",
    "    similarity_top_k=3,\n",
    "    # here we can control how granular citation sources are, the default is 512\n",
    "    citation_chunk_size=256,\n",
    ")\n",
    "\n",
    "# For citations we get the document info\n",
    "DB_DOC_ID_KEY = \"db_document_id\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. Prompt and Response function\n",
    "\n",
    "Pass in a question, get a response back.\n",
    "\n",
    "IMPORTANT: The prompt is set here, adjust it to match what you want the LLM to act like and do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def RunQuestion(questionText):\n",
    "    systemmessage = \"You are a story teller who likes to elaborate. Answer questions in a positive, helpful and interesting way. If the answer is not in the following context return ONLY 'Sorry, I don't know the answer to that'. Question: \"\n",
    "\n",
    "    queryQuestion = modelspecific_prompt(systemmessage, questionText)\n",
    "\n",
    "    response = query_engine.query(queryQuestion)\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 11. Questions to test with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "TestQuestions = [\n",
    "    \"Summarise the story for me\",\n",
    "    \"Who was the main protagonist?\",\n",
    "    \"Did they have any children? If so, what were their names?\",\n",
    "    \"Did anything eventful happen?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12. Run Questions through model (this can take a while) and see citations\n",
    "\n",
    "Runs each test question, saves it to a dictionary for output in the last step.\n",
    "\n",
    "Note: Citations are the source documents used and the text the response is based on. This is important for RAG so you can reference these documents for the user, and to ensure it's utilising the right documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1/4: Summarise the story for me\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    3898.61 ms\n",
      "llama_print_timings:      sample time =     101.09 ms /   343 runs   (    0.29 ms per token,  3393.05 tokens per second)\n",
      "llama_print_timings: prompt eval time =    5787.67 ms /  1191 tokens (    4.86 ms per token,   205.78 tokens per second)\n",
      "llama_print_timings:        eval time =   37533.99 ms /   342 runs   (  109.75 ms per token,     9.11 tokens per second)\n",
      "llama_print_timings:       total time =   44251.36 ms /  1533 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/3: |Thundertooth Part 1.docx| Source 1:\n",
      "Thundertooth faced a dilemma – he was hungry, but he couldn't bring himself to feast on the humans who scurried around like ants. As his hunger grew, he stumbled upon a park, an oasis of greenery amidst the concrete and steel. The park was adorned with holographic flowers that emitted a gentle glow, creating an ethereal atmosphere.\n",
      "\n",
      "\n",
      "\n",
      "While Thundertooth marveled at the beauty of the park, the mayor of the city happened to be passing by. Mayor Eleanor Grace, a charismatic and forward-thinking leader, was immediately intrigued by the sight of the talking dinosaur. She approached Thundertooth with a mix of curiosity and caution.\n",
      "\n",
      "\n",
      "\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "2/3: |Thundertooth Part 1.docx| Source 2:\n",
      "While Thundertooth marveled at the beauty of the park, the mayor of the city happened to be passing by. Mayor Eleanor Grace, a charismatic and forward-thinking leader, was immediately intrigued by the sight of the talking dinosaur. She approached Thundertooth with a mix of curiosity and caution.\n",
      "\n",
      "\n",
      "\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "\n",
      "\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "\n",
      "3/3: |Thundertooth Part 3.docx| Source 3:\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "\n",
      "2/4: Who was the main protagonist?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    3898.61 ms\n",
      "llama_print_timings:      sample time =      44.26 ms /   149 runs   (    0.30 ms per token,  3366.55 tokens per second)\n",
      "llama_print_timings: prompt eval time =    3082.38 ms /  1070 tokens (    2.88 ms per token,   347.13 tokens per second)\n",
      "llama_print_timings:        eval time =   15746.58 ms /   148 runs   (  106.40 ms per token,     9.40 tokens per second)\n",
      "llama_print_timings:       total time =   19216.33 ms /  1218 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/3: |Thundertooth Part 3.docx| Source 1:\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "2/3: |Thundertooth Part 3.docx| Source 2:\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth stood at the forefront, using his mighty roar to coordinate and inspire the efforts of the city's inhabitants. The ground trembled as the meteor drew closer, but the Thundertooth family's coordinated efforts began to take effect. Lumina's force field shimmered to life, deflecting the meteor's deadly path. Echo's amplified warnings reached every corner of the city, ensuring that no one was left behind.\n",
      "\n",
      "3/3: |Thundertooth Part 3.docx| Source 3:\n",
      "The mayor, recognizing Thundertooth's intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "3/4: Did they have any children? If so, what were their names?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    3898.61 ms\n",
      "llama_print_timings:      sample time =      33.35 ms /   112 runs   (    0.30 ms per token,  3357.92 tokens per second)\n",
      "llama_print_timings: prompt eval time =    1959.82 ms /   885 tokens (    2.21 ms per token,   451.57 tokens per second)\n",
      "llama_print_timings:        eval time =   11779.25 ms /   111 runs   (  106.12 ms per token,     9.42 tokens per second)\n",
      "llama_print_timings:       total time =   14020.71 ms /   996 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/3: |Thundertooth Part 2.docx| Source 1:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "2/3: |Thundertooth Part 2.docx| Source 2:\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "\n",
      "Echo: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\n",
      "\n",
      "3/3: |Thundertooth Part 2.docx| Source 3:\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "4/4: Did anything eventful happen?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    3898.61 ms\n",
      "llama_print_timings:      sample time =      91.72 ms /   310 runs   (    0.30 ms per token,  3380.00 tokens per second)\n",
      "llama_print_timings: prompt eval time =    2095.75 ms /   961 tokens (    2.18 ms per token,   458.55 tokens per second)\n",
      "llama_print_timings:        eval time =   33070.89 ms /   309 runs   (  107.03 ms per token,     9.34 tokens per second)\n",
      "llama_print_timings:       total time =   35991.08 ms /  1270 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/3: |Thundertooth Part 3.docx| Source 1:\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "2/3: |Thundertooth Part 1.docx| Source 2:\n",
      "While Thundertooth marveled at the beauty of the park, the mayor of the city happened to be passing by. Mayor Eleanor Grace, a charismatic and forward-thinking leader, was immediately intrigued by the sight of the talking dinosaur. She approached Thundertooth with a mix of curiosity and caution.\n",
      "\n",
      "\n",
      "\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "\n",
      "\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "\n",
      "3/3: |Thundertooth Part 1.docx| Source 3:\n",
      "Thundertooth faced a dilemma – he was hungry, but he couldn't bring himself to feast on the humans who scurried around like ants. As his hunger grew, he stumbled upon a park, an oasis of greenery amidst the concrete and steel. The park was adorned with holographic flowers that emitted a gentle glow, creating an ethereal atmosphere.\n",
      "\n",
      "\n",
      "\n",
      "While Thundertooth marveled at the beauty of the park, the mayor of the city happened to be passing by. Mayor Eleanor Grace, a charismatic and forward-thinking leader, was immediately intrigued by the sight of the talking dinosaur. She approached Thundertooth with a mix of curiosity and caution.\n",
      "\n",
      "\n",
      "\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "qa_pairs = []\n",
    "\n",
    "for index, question in enumerate(TestQuestions, start=1):\n",
    "    question = question.strip() # Clean up\n",
    "\n",
    "    print(f\"\\n{index}/{len(TestQuestions)}: {question}\")\n",
    "\n",
    "    response = RunQuestion(question) # Query and get  response\n",
    "\n",
    "    qa_pairs.append((question.strip(), str(response).strip())) # Add to our output array\n",
    "\n",
    "    # Displays the citations\n",
    "    for index, node in enumerate(response.source_nodes, start=1):\n",
    "        print(f\"{index}/{len(response.source_nodes)}: |{node.node.metadata['file_name']}| {node.node.get_text()}\")\n",
    "\n",
    "    # Uncomment the following line if you want to test just the first question\n",
    "    # break "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 13. Output responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4 Summarise the story for me\n",
      "\n",
      "In this captivating tale, we encounter Thundertooth, a time-traveling dinosaur who finds himself in the future, grappling with the dilemma of his predatory nature towards humans. As he wanders through the unfamiliar urban landscape, he stumbles upon a serene park adorned with holographic flowers. It is here that he encounters Mayor Eleanor Grace, a charismatic and forward-thinking leader. Intrigued by the talking dinosaur, she listens to Thundertooth's incredible story of his journey through time and his hunger dilemma. Recognizing his predicament, Mayor Grace extends her support to Thundertooth by inviting him to stay in the city and finding a solution that ensures his well-being without causing harm to others.\n",
      "\n",
      "In another part of the story, we meet Thundertooth alongside his companions Lumina, Echo, Sapphire, and Ignis, who collectively face another challenge - an incoming meteor threatening the city's safety. Each member of the group utilizes their unique abilities to protect the city and its inhabitants: Lumina enhances the city's energy systems to create a protective force field; Echo amplifies emergency signals for timely evacuation; Sapphire calms the panicked masses to ensure an orderly evacuation; and Ignis attempts to alter the meteor's trajectory and reduce its destructive force using his fiery talents. Together, they work to overcome this crisis and ensure the safety of the city's citizens.\n",
      "\n",
      "--------\n",
      "\n",
      "2/4 Who was the main protagonist?\n",
      "\n",
      "In the provided sources, the main protagonist seems to be Thundertooth. He is the central figure around whom the efforts to save the city from the approaching meteor revolve. Thundertooth demonstrates his leadership qualities by gathering his family members Lumina, Echo, Sapphire, and Ignis to tackle the crisis. Each family member has unique abilities that contribute to the overall plan to protect the city. Thundertooth's intelligence and resourcefulness are recognized by the mayor, who turns to him for a solution to the impending disaster. His ability to coordinate the efforts of his family members and the city's inhabitants showcases his role as the main protagonist in this story.\n",
      "\n",
      "--------\n",
      "\n",
      "3/4 Did they have any children? If so, what were their names?\n",
      "\n",
      "Yes, Thundertooth and his intelligent dinosaur companion Seraphina had a beautiful family together. They were blessed with four unique children who embodied the diversity of their modern world. Their names are: Lumina, the eldest who inherited her parents' remarkable traits; Echo, the second-born with the gift of mimicry; details about their other two children aren't explicitly mentioned in the provided sources, but it's clear that their family life flourished alongside Thundertooth's successful toy factory.\n",
      "\n",
      "--------\n",
      "\n",
      "4/4 Did anything eventful happen?\n",
      "\n",
      "Indeed, several eventful occurrences took place in the provided context. To elaborate, our main character Thundertooth, the time-traveling dinosaur, found himself in an unfamiliar world filled with advanced technology and human civilization. This sudden shift from his prehistoric environment to the futuristic city presented numerous challenges for him, particularly his hunger dilemma - unable to consume humans but still feeling the pangs of hunger.\n",
      "\n",
      "While exploring this new world, Thundertooth stumbled upon a park where he caught the attention of Mayor Eleanor Grace. Their encounter led to an intriguing conversation about Thundertooth's journey through time and his predicament in the modern world. This meeting proved to be pivotal, as Mayor Grace welcomed Thundertooth into the city and promised to find a solution for his hunger problem without causing harm to anyone.\n",
      "\n",
      "Later on, another eventful situation unfolded when the city faced a crisis - the imminent threat of a meteor impact. Thundertooth's family, along with Lumina, Echo, Sapphire, and Ignis, sprang into action to protect the city's inhabitants by strengthening the energy systems, amplifying emergency signals, calming the masses, and attempting to alter the meteor's trajectory. These events showcase the resilience of both Thundertooth and the human population in the face of adversity.\n",
      "\n",
      "--------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, (question, answer) in enumerate(qa_pairs, start=1):\n",
    "    print(f\"{index}/{len(qa_pairs)} {question}\\n\\n{answer}\\n\\n--------\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llamaindexgeneric",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}