18.Convert_Document_to_Knowledge_Graph_Langchain_Openai.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Constructing and Retriving Information From KnowledgeGraph using Langchain\n",
    "\n",
    "# 1.Introduction\n",
    "\n",
    "* Graph Retrieval Augemented Generation[Graph RAG](https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/?ref=blog.langchain.dev) is gaining momentum and emerging as powerful additional traditional vector search retrieval methods.\n",
    "* This approach leverages the structured nature of graph databases, which organize data as nodoe and relationships, to enhance the depth and cotnexuality retrieved information.\n",
    "\n",
    "![](https://miro.medium.com/v2/resize:fit:700/0*rm_fRSPovV1wfTqH.jpg)\n",
    "\n",
    "* Graphs are great at representing and storing heterogeneous and interconnected information in a structured manner, effortlessly capturing complex relationships and attributes across diverse data types.\n",
    "* In contrast, vector databases often struggle with such structured information, as their strength lies in handling unstructured data through high-dimensional vectors.\n",
    "* In your RAG application, you can combine structured graph data with vector search through unstructured text to achieve the best of both worlds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.Knowledge graphs are great, but how do you create one?\n",
    "\n",
    "* Constructing a knowledge graph is typically the most challenging step in leveraging the power of graph-based data representation.\n",
    "* It involves gathering and structuring the data, which requires a deep understanding of both the domain and graph modeling. \n",
    "* To simplify this process, we have been experimenting with LLMs. LLMs, with their profound understanding of language and context, can automate significant parts of the knowledge graph creation process. \n",
    "* By analyzing text data, these models can identify entities, understand the relationships between them, and suggest how they might be best represented in a graph structure. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.Installation of Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install --upgrade --quiet  langchain langchain-community langchain-openai langchain-experimental neo4j wikipedia tiktoken yfiles_jupyter_graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.Import Required Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.runnables import (\n",
    "    RunnableBranch,\n",
    "    RunnableLambda,\n",
    "    RunnableParallel,\n",
    "    RunnablePassthrough,\n",
    ")\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.prompts.prompt import PromptTemplate\n",
    "from langchain_core.pydantic_v1 import BaseModel, Field\n",
    "from typing import Tuple, List, Optional\n",
    "from langchain_core.messages import AIMessage, HumanMessage\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "import os\n",
    "from langchain_community.graphs import Neo4jGraph\n",
    "from langchain.document_loaders import WikipediaLoader\n",
    "from langchain.text_splitter import TokenTextSplitter\n",
    "from langchain_openai import ChatOpenAI\n",
    "from langchain_experimental.graph_transformers import LLMGraphTransformer\n",
    "from neo4j import GraphDatabase\n",
    "from yfiles_jupyter_graphs import GraphWidget\n",
    "from langchain_community.vectorstores import Neo4jVector\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_community.vectorstores.neo4j_vector import remove_lucene_chars\n",
    "from langchain_core.runnables import ConfigurableField, RunnableParallel, RunnablePassthrough"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5.Configue the Neo4j Environment Setup:\n",
    "\n",
    "* You need to set up a Neo4j instance follow along with the examples in this blog post. The easiest way is to start a free instance on [Neo4j Aura](https://neo4j.com/cloud/platform/aura-graph-database/), which offers cloud instances of Neo4j database. Alternatively, you can also set up a local instance of the Neo4j database by downloading the Neo4j Desktop application and creating a local database instance.\n",
    "\n",
    "* **Note : You can also connect your local database as well I am using Local Neo4j DB to connect.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-\"\n",
    "os.environ[\"NEO4J_URI\"] = \"bolt://localhost:7687\"\n",
    "os.environ[\"NEO4J_USERNAME\"] = \"neo4j\"\n",
    "os.environ[\"NEO4J_PASSWORD\"] = \"admin@123\"\n",
    "\n",
    "graph = Neo4jGraph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.DataInjection Pipeline\n",
    "\n",
    "- we will use LLMs I's Wikipedia page. We can utilize LangChain loaders to fetch and split the documents from Wikipedia seamlessly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the wikipedia\n",
    "raw_documents = WikipediaLoader(query=\"Large Language Model\").load()\n",
    "\n",
    "# Define chunking Strategy\n",
    "text_splitter = TokenTextSplitter(chunk_size=512, chunk_overlap = 24)\n",
    "documents = text_splitter.split_documents(raw_documents[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content=\"India, officially the Republic of India (ISO: Bhārat Gaṇarājya), is a country in South Asia.  It is the seventh-largest country by area; the most populous country as of June 2023; and from the time of its independence in 1947, the world's most populous democracy. Bounded by the Indian Ocean on the south, the Arabian Sea on the southwest, and the Bay of Bengal on the southeast, it shares land borders with Pakistan to the west; China, Nepal, and Bhutan to the north; and Bangladesh and Myanmar to the east. In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives; its Andaman and Nicobar Islands share a maritime border with Thailand, Myanmar, and Indonesia.\\nModern humans arrived on the Indian subcontinent from Africa no later than 55,000 years ago.\\nTheir long occupation, initially in varying forms of isolation as hunter-gatherers, has made the region highly diverse, second only to Africa in human genetic diversity. Settled life emerged on the subcontinent in the western margins of the Indus river basin 9,000 years ago, evolving gradually into the Indus Valley Civilisation of the third millennium BCE.\\nBy 1200 BCE, an archaic form of Sanskrit, an Indo-European language, had diffused into India from the northwest. Its evidence today is found in the hymns of the Rigveda. Preserved by an oral tradition that was resolutely vigilant, the Rigveda records the dawning of Hinduism in India. The Dravidian languages of India were supplanted in the northern and western regions.\\nBy 400 BCE, stratification and exclusion by caste had emerged within Hinduism,\\nand Buddhism and Jainism had arisen, proclaiming social orders unlinked to heredity.\\nEarly political consolidations gave rise to the loose-knit Maurya and Gupta Empires based in the Ganges Basin.\\nTheir collective era was suffused with wide-ranging creativity, but also marked by the declining status of women, and the incorporation of untouchability into an organised system of belief. In South India, the Middle kingdoms exported Dravidian-languages scripts and religious cultures to the kingdoms of Southeast Asia.In the early medieval era, Christianity, Islam, Judaism, and Zoroastrianism became established on India's southern and western coasts.\\nMuslim armies from Central Asia intermittently overran India's northern plains,\\neventually founding the Delhi Sultanate\", metadata={'title': 'India', 'summary': \"India, officially the Republic of India (ISO: Bhārat Gaṇarājya), is a country in South Asia.  It is the seventh-largest country by area; the most populous country as of June 2023; and from the time of its independence in 1947, the world's most populous democracy. Bounded by the Indian Ocean on the south, the Arabian Sea on the southwest, and the Bay of Bengal on the southeast, it shares land borders with Pakistan to the west; China, Nepal, and Bhutan to the north; and Bangladesh and Myanmar to the east. In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives; its Andaman and Nicobar Islands share a maritime border with Thailand, Myanmar, and Indonesia.\\nModern humans arrived on the Indian subcontinent from Africa no later than 55,000 years ago.\\nTheir long occupation, initially in varying forms of isolation as hunter-gatherers, has made the region highly diverse, second only to Africa in human genetic diversity. Settled life emerged on the subcontinent in the western margins of the Indus river basin 9,000 years ago, evolving gradually into the Indus Valley Civilisation of the third millennium BCE.\\nBy 1200 BCE, an archaic form of Sanskrit, an Indo-European language, had diffused into India from the northwest. Its evidence today is found in the hymns of the Rigveda. Preserved by an oral tradition that was resolutely vigilant, the Rigveda records the dawning of Hinduism in India. The Dravidian languages of India were supplanted in the northern and western regions.\\nBy 400 BCE, stratification and exclusion by caste had emerged within Hinduism,\\nand Buddhism and Jainism had arisen, proclaiming social orders unlinked to heredity.\\nEarly political consolidations gave rise to the loose-knit Maurya and Gupta Empires based in the Ganges Basin.\\nTheir collective era was suffused with wide-ranging creativity, but also marked by the declining status of women, and the incorporation of untouchability into an organised system of belief. In South India, the Middle kingdoms exported Dravidian-languages scripts and religious cultures to the kingdoms of Southeast Asia.In the early medieval era, Christianity, Islam, Judaism, and Zoroastrianism became established on India's southern and western coasts.\\nMuslim armies from Central Asia intermittently overran India's northern plains,\\neventually founding the Delhi Sultanate, and drawing northern India into the cosmopolitan networks of medieval Islam.\\nIn the 15th century, the Vijayanagara Empire created a long-lasting composite Hindu culture in south India.\\nIn the Punjab, Sikhism emerged, rejecting institutionalised religion.\\nThe Mughal Empire, in 1526, ushered in two centuries of relative peace,\\nleaving a legacy of luminous architecture.\\nGradually expanding rule of the British East India Company followed, turning India into a colonial economy, but also consolidating its sovereignty. British Crown rule began in 1858. The rights promised to Indians were granted slowly, but technological changes were introduced, and modern ideas of education and the public life took root. A pioneering and influential nationalist movement emerged, which was noted for nonviolent resistance and became the major factor in ending British rule. In 1947 the British Indian Empire was partitioned into two independent dominions, a Hindu-majority Dominion of India and a Muslim-majority Dominion of Pakistan, amid large-scale loss of life and an unprecedented migration.India has been a federal republic since 1950, governed through a democratic parliamentary system. It is a pluralistic, multilingual and multi-ethnic society. India's population grew from 361 million in 1951 to almost 1.4 billion in 2022.\\nDuring the same time, its nominal per capita income increased from US$64 annually to US$2,601, and its literacy rate from 16.6% to 74%. From being a comparatively destitute country in 1951, India has become a fast-growing major economy and a hub for information technology services, with an expanding middle class. India has a space programme with several planned or completed extraterrestrial missions. Indian movies, music, and spiritual teachings play an increasing role in global culture. India has substantially reduced its rate of poverty, though at the cost of increasing economic inequality. India is a nuclear-weapon state, which ranks high in military expenditure. It has disputes over Kashmir with its neighbours, Pakistan and China, unresolved since the mid-20th century. Among the socio-economic challenges India faces are gender inequality, child malnutrition, and rising levels of air pollution. India's land is megadiverse, with four biodiversity hotspots. Its forest cover comprises 21.7% of its area. India's wildlife, which has traditionally been viewed with tolerance in India's culture, is supported among these forests, and elsewhere, in protected habitats.\", 'source': 'https://en.wikipedia.org/wiki/India'}),\n",
       " Document(page_content=\".\\nMuslim armies from Central Asia intermittently overran India's northern plains,\\neventually founding the Delhi Sultanate, and drawing northern India into the cosmopolitan networks of medieval Islam.\\nIn the 15th century, the Vijayanagara Empire created a long-lasting composite Hindu culture in south India.\\nIn the Punjab, Sikhism emerged, rejecting institutionalised religion.\\nThe Mughal Empire, in 1526, ushered in two centuries of relative peace,\\nleaving a legacy of luminous architecture.\\nGradually expanding rule of the British East India Company followed, turning India into a colonial economy, but also consolidating its sovereignty. British Crown rule began in 1858. The rights promised to Indians were granted slowly, but technological changes were introduced, and modern ideas of education and the public life took root. A pioneering and influential nationalist movement emerged, which was noted for nonviolent resistance and became the major factor in ending British rule. In 1947 the British Indian Empire was partitioned into two independent dominions, a Hindu-majority Dominion of India and a Muslim-majority Dominion of Pakistan, amid large-scale loss of life and an unprecedented migration.India has been a federal republic since 1950, governed through a democratic parliamentary system. It is a pluralistic, multilingual and multi-ethnic society. India's population grew from 361 million in 1951 to almost 1.4 billion in 2022.\\nDuring the same time, its nominal per capita income increased from US$64 annually to US$2,601, and its literacy rate from 16.6% to 74%. From being a comparatively destitute country in 1951, India has become a fast-growing major economy and a hub for information technology services, with an expanding middle class. India has\", metadata={'title': 'India', 'summary': \"India, officially the Republic of India (ISO: Bhārat Gaṇarājya), is a country in South Asia.  It is the seventh-largest country by area; the most populous country as of June 2023; and from the time of its independence in 1947, the world's most populous democracy. Bounded by the Indian Ocean on the south, the Arabian Sea on the southwest, and the Bay of Bengal on the southeast, it shares land borders with Pakistan to the west; China, Nepal, and Bhutan to the north; and Bangladesh and Myanmar to the east. In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives; its Andaman and Nicobar Islands share a maritime border with Thailand, Myanmar, and Indonesia.\\nModern humans arrived on the Indian subcontinent from Africa no later than 55,000 years ago.\\nTheir long occupation, initially in varying forms of isolation as hunter-gatherers, has made the region highly diverse, second only to Africa in human genetic diversity. Settled life emerged on the subcontinent in the western margins of the Indus river basin 9,000 years ago, evolving gradually into the Indus Valley Civilisation of the third millennium BCE.\\nBy 1200 BCE, an archaic form of Sanskrit, an Indo-European language, had diffused into India from the northwest. Its evidence today is found in the hymns of the Rigveda. Preserved by an oral tradition that was resolutely vigilant, the Rigveda records the dawning of Hinduism in India. The Dravidian languages of India were supplanted in the northern and western regions.\\nBy 400 BCE, stratification and exclusion by caste had emerged within Hinduism,\\nand Buddhism and Jainism had arisen, proclaiming social orders unlinked to heredity.\\nEarly political consolidations gave rise to the loose-knit Maurya and Gupta Empires based in the Ganges Basin.\\nTheir collective era was suffused with wide-ranging creativity, but also marked by the declining status of women, and the incorporation of untouchability into an organised system of belief. In South India, the Middle kingdoms exported Dravidian-languages scripts and religious cultures to the kingdoms of Southeast Asia.In the early medieval era, Christianity, Islam, Judaism, and Zoroastrianism became established on India's southern and western coasts.\\nMuslim armies from Central Asia intermittently overran India's northern plains,\\neventually founding the Delhi Sultanate, and drawing northern India into the cosmopolitan networks of medieval Islam.\\nIn the 15th century, the Vijayanagara Empire created a long-lasting composite Hindu culture in south India.\\nIn the Punjab, Sikhism emerged, rejecting institutionalised religion.\\nThe Mughal Empire, in 1526, ushered in two centuries of relative peace,\\nleaving a legacy of luminous architecture.\\nGradually expanding rule of the British East India Company followed, turning India into a colonial economy, but also consolidating its sovereignty. British Crown rule began in 1858. The rights promised to Indians were granted slowly, but technological changes were introduced, and modern ideas of education and the public life took root. A pioneering and influential nationalist movement emerged, which was noted for nonviolent resistance and became the major factor in ending British rule. In 1947 the British Indian Empire was partitioned into two independent dominions, a Hindu-majority Dominion of India and a Muslim-majority Dominion of Pakistan, amid large-scale loss of life and an unprecedented migration.India has been a federal republic since 1950, governed through a democratic parliamentary system. It is a pluralistic, multilingual and multi-ethnic society. India's population grew from 361 million in 1951 to almost 1.4 billion in 2022.\\nDuring the same time, its nominal per capita income increased from US$64 annually to US$2,601, and its literacy rate from 16.6% to 74%. From being a comparatively destitute country in 1951, India has become a fast-growing major economy and a hub for information technology services, with an expanding middle class. India has a space programme with several planned or completed extraterrestrial missions. Indian movies, music, and spiritual teachings play an increasing role in global culture. India has substantially reduced its rate of poverty, though at the cost of increasing economic inequality. India is a nuclear-weapon state, which ranks high in military expenditure. It has disputes over Kashmir with its neighbours, Pakistan and China, unresolved since the mid-20th century. Among the socio-economic challenges India faces are gender inequality, child malnutrition, and rising levels of air pollution. India's land is megadiverse, with four biodiversity hotspots. Its forest cover comprises 21.7% of its area. India's wildlife, which has traditionally been viewed with tolerance in India's culture, is supported among these forests, and elsewhere, in protected habitats.\", 'source': 'https://en.wikipedia.org/wiki/India'}),\n",
       " Document(page_content=\"The economy of India has transitioned from a mixed planned economy to a mixed middle-income developing social market economy with notable public sector in strategic sectors. It is the world's fifth-largest economy by nominal GDP and the third-largest by purchasing power parity (PPP); on a per capita income basis, India ranked 139th by GDP (nominal) and 127th by GDP (PPP). From independence in 1947 until 1991, successive governments followed Soviet model and promoted protectionist economic policies, with extensive Sovietization, state intervention, demand-side economics, natural resources, bureaucrat driven enterprises and economic regulation. This is characterised as dirigism, in the form of the Licence Raj. The end of the Cold War and an acute balance of payments crisis in 1991 led to the adoption of a broad economic liberalisation in India and indicative planning. Since the start of the 21st century, annual average GDP growth has been 6% to 7%. The economy of the Indian subcontinent was the largest in the world for most of recorded history up until the onset of colonialism in early 19th century.Nearly 70% of India's GDP is driven by domestic consumption; country remains the world's sixth-largest consumer market. Apart from private consumption, India's GDP is also fueled by government spending, investments, and exports. In 2022, India was the world's 8th-largest importer and the 10th-largest exporter. India has been a member of the World Trade Organization since 1 January 1995. It ranks 63rd on the Ease of doing business index and 40th on the Global Competitiveness Index. With 476 million workers, the Indian labour force is the world's second-largest. India has one of the world's highest number of billionaires and extreme income inequality.During the 2008 global financial crisis, the economy faced a mild slowdown. India endorsed Keynesian policy and initiated stimulus measures (both fiscal and monetary) to boost growth and generate demand. In subsequent years, economic growth revived. The period between 2004 and 2014 is referred to as India's lost decade as India fell behind other BRIC economies.In 2021–22, the foreign direct investment (FDI) in India was $82 billion. The leading sectors for FDI inflows were the service sector, the computer industry, and the telecom industry. India has free trade agreements with several nations and blocs, including ASEAN, SAFTA, Mercosur, South Korea, Japan, Australia, UAE, and several others which are in effect or under\", metadata={'title': 'Economy of India', 'summary': \"The economy of India has transitioned from a mixed planned economy to a mixed middle-income developing social market economy with notable public sector in strategic sectors. It is the world's fifth-largest economy by nominal GDP and the third-largest by purchasing power parity (PPP); on a per capita income basis, India ranked 139th by GDP (nominal) and 127th by GDP (PPP). From independence in 1947 until 1991, successive governments followed Soviet model and promoted protectionist economic policies, with extensive Sovietization, state intervention, demand-side economics, natural resources, bureaucrat driven enterprises and economic regulation. This is characterised as dirigism, in the form of the Licence Raj. The end of the Cold War and an acute balance of payments crisis in 1991 led to the adoption of a broad economic liberalisation in India and indicative planning. Since the start of the 21st century, annual average GDP growth has been 6% to 7%. The economy of the Indian subcontinent was the largest in the world for most of recorded history up until the onset of colonialism in early 19th century.Nearly 70% of India's GDP is driven by domestic consumption; country remains the world's sixth-largest consumer market. Apart from private consumption, India's GDP is also fueled by government spending, investments, and exports. In 2022, India was the world's 8th-largest importer and the 10th-largest exporter. India has been a member of the World Trade Organization since 1 January 1995. It ranks 63rd on the Ease of doing business index and 40th on the Global Competitiveness Index. With 476 million workers, the Indian labour force is the world's second-largest. India has one of the world's highest number of billionaires and extreme income inequality.During the 2008 global financial crisis, the economy faced a mild slowdown. India endorsed Keynesian policy and initiated stimulus measures (both fiscal and monetary) to boost growth and generate demand. In subsequent years, economic growth revived. The period between 2004 and 2014 is referred to as India's lost decade as India fell behind other BRIC economies.In 2021–22, the foreign direct investment (FDI) in India was $82 billion. The leading sectors for FDI inflows were the service sector, the computer industry, and the telecom industry. India has free trade agreements with several nations and blocs, including ASEAN, SAFTA, Mercosur, South Korea, Japan, Australia, UAE, and several others which are in effect or under negotiating stage.The service sector makes up  more than 50% of GDP and remains the fastest growing sector, while the industrial sector and the agricultural sector employs a majority of the labor force. The Bombay Stock Exchange and National Stock Exchange are some of the world's largest stock exchanges by market capitalisation. India is the world's sixth-largest manufacturer, representing 2.6% of global manufacturing output. Nearly 65% of India's population is rural, and contributes about 50% of India's GDP. India faces high unemployment, rising income inequality, and a drop in aggregate demand. India's gross domestic savings rate stood at 29.3% of GDP in 2022. In recent years, independent economists and financial institutions have accused the government of manipulating various economic data, especially GDP growth. India's overall social spending as a share of GDP in 2021–22 will be 8.6%, which is much lower than the average for OECD nations.\", 'source': 'https://en.wikipedia.org/wiki/Economy_of_India'}),\n",
       " Document(page_content=\"AFTA, Mercosur, South Korea, Japan, Australia, UAE, and several others which are in effect or under negotiating stage.The service sector makes up  more than 50% of GDP and remains the fastest growing sector, while the industrial sector and the agricultural sector employs a majority of the labor force. The Bombay Stock Exchange and National Stock Exchange are some of the world's largest stock exchanges by market capitalisation. India is the world's sixth-largest manufacturer, representing 2.6% of global manufacturing output. Nearly 65% of India's population is rural, and contributes about 50% of India's GDP. India faces high unemployment, rising income inequality, and a drop in aggregate demand. India's gross domestic savings rate stood at 29.3% of GDP in 2022. In recent years, independent economists and financial institutions have accused the government of manipulating various economic data, especially GDP growth. India's overall social spending as a share of GDP in 2021–22 will be 8.6%, which is much lower than the average for OECD nations.\\n\\n\\n== History ==\\n\\nFor a continuous duration of nearly 1700 years from the year 1 CE, India was the world's largest economy, constituting 35 to 40% of the world GDP. The combination of protectionist, import-substitution, Fabian socialism, and social democratic-inspired policies governed India for sometime after the end of British rule. The economy was then characterised as Dirigism, It had extensive regulation, protectionism, public ownership of large monopolies, pervasive corruption and slow growth. Since 1991, continuing economic libera\", metadata={'title': 'Economy of India', 'summary': \"The economy of India has transitioned from a mixed planned economy to a mixed middle-income developing social market economy with notable public sector in strategic sectors. It is the world's fifth-largest economy by nominal GDP and the third-largest by purchasing power parity (PPP); on a per capita income basis, India ranked 139th by GDP (nominal) and 127th by GDP (PPP). From independence in 1947 until 1991, successive governments followed Soviet model and promoted protectionist economic policies, with extensive Sovietization, state intervention, demand-side economics, natural resources, bureaucrat driven enterprises and economic regulation. This is characterised as dirigism, in the form of the Licence Raj. The end of the Cold War and an acute balance of payments crisis in 1991 led to the adoption of a broad economic liberalisation in India and indicative planning. Since the start of the 21st century, annual average GDP growth has been 6% to 7%. The economy of the Indian subcontinent was the largest in the world for most of recorded history up until the onset of colonialism in early 19th century.Nearly 70% of India's GDP is driven by domestic consumption; country remains the world's sixth-largest consumer market. Apart from private consumption, India's GDP is also fueled by government spending, investments, and exports. In 2022, India was the world's 8th-largest importer and the 10th-largest exporter. India has been a member of the World Trade Organization since 1 January 1995. It ranks 63rd on the Ease of doing business index and 40th on the Global Competitiveness Index. With 476 million workers, the Indian labour force is the world's second-largest. India has one of the world's highest number of billionaires and extreme income inequality.During the 2008 global financial crisis, the economy faced a mild slowdown. India endorsed Keynesian policy and initiated stimulus measures (both fiscal and monetary) to boost growth and generate demand. In subsequent years, economic growth revived. The period between 2004 and 2014 is referred to as India's lost decade as India fell behind other BRIC economies.In 2021–22, the foreign direct investment (FDI) in India was $82 billion. The leading sectors for FDI inflows were the service sector, the computer industry, and the telecom industry. India has free trade agreements with several nations and blocs, including ASEAN, SAFTA, Mercosur, South Korea, Japan, Australia, UAE, and several others which are in effect or under negotiating stage.The service sector makes up  more than 50% of GDP and remains the fastest growing sector, while the industrial sector and the agricultural sector employs a majority of the labor force. The Bombay Stock Exchange and National Stock Exchange are some of the world's largest stock exchanges by market capitalisation. India is the world's sixth-largest manufacturer, representing 2.6% of global manufacturing output. Nearly 65% of India's population is rural, and contributes about 50% of India's GDP. India faces high unemployment, rising income inequality, and a drop in aggregate demand. India's gross domestic savings rate stood at 29.3% of GDP in 2022. In recent years, independent economists and financial institutions have accused the government of manipulating various economic data, especially GDP growth. India's overall social spending as a share of GDP in 2021–22 will be 8.6%, which is much lower than the average for OECD nations.\", 'source': 'https://en.wikipedia.org/wiki/Economy_of_India'}),\n",
       " Document(page_content=\"Punjab ( ; Punjabi: [pənˈdʒɑːb]), historically known as Panchanada or Pentapotamia, is a state in northern India. Forming part of the larger Punjab region of the Indian subcontinent, the state is bordered by the Indian states of Himachal Pradesh to the north and northeast, Haryana to the south and southeast, and Rajasthan to the southwest; by the Indian union territories of Chandigarh to the east and Jammu and Kashmir to the north. It shares an international border with Punjab, a province of Pakistan to the west. The state covers an area of 50,362 square kilometres (19,445 square miles), which is 1.53% of India's total geographical area, making it the 19th-largest Indian state by area out of 28 Indian states (20th largest, if Union Territories are considered). With over 27 million inhabitants, Punjab is the 16th-largest Indian state by population, comprising 23 districts. Punjabi, written in the Gurmukhi script, is the most widely spoken and the official language of the state. The main ethnic group are the Punjabis, with Sikhs (57.7%) and Hindus (38.5%) forming the dominant religious groups. The state capital, Chandigarh, is a union territory and also the capital of the neighbouring state of Haryana. Three tributaries of the Indus River — the Sutlej, Beas, and Ravi — flow through Punjab.The history of Punjab has witnessed the migration and settlement of different tribes of people with different cultures and ideas, forming a civilisational melting pot. The ancient Indus Valley civilisation flourished in the region until its decline around 1900 BCE. Punjab was enriched during the height of the Vedic period, but declined in predominance with the rise of the Mahajanapadas. The region formed the frontier of initial empires during antiquity including Alexander's and the Maurya empires. It was subsequently conquered by the Kushan Empire, Gupta Empire, and then Harsha's Empire. Punjab continued to be settled by nomadic people; including the Huna, Turkic and the Mongols. Punjab came under Muslim rule c.\\u20091000 CE, and was part of the Delhi Sultanate and the Mughal Empire. Sikhism, based on the teachings of Sikh Gurus, emerged between the 15th and 17th centuries. Conflicts between the\", metadata={'title': 'Punjab, India', 'summary': \"Punjab ( ; Punjabi: [pənˈdʒɑːb]), historically known as Panchanada or Pentapotamia, is a state in northern India. Forming part of the larger Punjab region of the Indian subcontinent, the state is bordered by the Indian states of Himachal Pradesh to the north and northeast, Haryana to the south and southeast, and Rajasthan to the southwest; by the Indian union territories of Chandigarh to the east and Jammu and Kashmir to the north. It shares an international border with Punjab, a province of Pakistan to the west. The state covers an area of 50,362 square kilometres (19,445 square miles), which is 1.53% of India's total geographical area, making it the 19th-largest Indian state by area out of 28 Indian states (20th largest, if Union Territories are considered). With over 27 million inhabitants, Punjab is the 16th-largest Indian state by population, comprising 23 districts. Punjabi, written in the Gurmukhi script, is the most widely spoken and the official language of the state. The main ethnic group are the Punjabis, with Sikhs (57.7%) and Hindus (38.5%) forming the dominant religious groups. The state capital, Chandigarh, is a union territory and also the capital of the neighbouring state of Haryana. Three tributaries of the Indus River — the Sutlej, Beas, and Ravi — flow through Punjab.The history of Punjab has witnessed the migration and settlement of different tribes of people with different cultures and ideas, forming a civilisational melting pot. The ancient Indus Valley civilisation flourished in the region until its decline around 1900 BCE. Punjab was enriched during the height of the Vedic period, but declined in predominance with the rise of the Mahajanapadas. The region formed the frontier of initial empires during antiquity including Alexander's and the Maurya empires. It was subsequently conquered by the Kushan Empire, Gupta Empire, and then Harsha's Empire. Punjab continued to be settled by nomadic people; including the Huna, Turkic and the Mongols. Punjab came under Muslim rule c.\\u20091000 CE, and was part of the Delhi Sultanate and the Mughal Empire. Sikhism, based on the teachings of Sikh Gurus, emerged between the 15th and 17th centuries. Conflicts between the Mughals and the later Sikh Gurus precipitated a militarisation of the Sikhs, resulting in the formation of a confederacy after the weakening of the Mughal Empire, which competed for control with the larger Durrani Empire. This confederacy was united in 1801 by Maharaja Ranjit Singh, forming the Sikh Empire.The larger Punjab region was annexed by the British East India Company from the Sikh Empire in 1849. At the time of the independence of India from British rule in 1947, the Punjab province was partitioned along religious lines amidst widespread violence, with the Muslim-majority western portion becoming part of Pakistan and the Hindu- and Sikh-majority east remaining in India, causing a large-scale migration between the two. After the Punjabi Suba movement, Indian Punjab was reorganised on the basis of language in 1966, when its Haryanvi- and Hindi-speaking areas were carved out as Haryana, Pahari-speaking regions attached to Himachal Pradesh and the remaining, mostly Punjabi-speaking areas became the current state of Punjab. A separatist insurgency occurred in the state during the 1980s. At present, the economy of Punjab is the 15th-largest state economy in India with ₹5.29 trillion (US$66 billion) in gross domestic product and a per capita GDP of ₹151,367 (US$1,900), ranking 17th among Indian states. Since independence, Punjab is predominantly an agrarian society. It is the ninth-highest ranking among Indian states in human development index. Punjab has bustling tourism, music, culinary, and film industries.\", 'source': 'https://en.wikipedia.org/wiki/Punjab,_India'}),\n",
       " Document(page_content=', based on the teachings of Sikh Gurus, emerged between the 15th and 17th centuries. Conflicts between the Mughals and the later Sikh Gurus precipitated a militarisation of the Sikhs, resulting in the formation of a confederacy after the weakening of the Mughal Empire, which competed for control with the larger Durrani Empire. This confederacy was united in 1801 by Maharaja Ranjit Singh, forming the Sikh Empire.The larger Punjab region was annexed by the British East India Company from the Sikh Empire in 1849. At the time of the independence of India from British rule in 1947, the Punjab province was partitioned along religious lines amidst widespread violence, with the Muslim-majority western portion becoming part of Pakistan and the Hindu- and Sikh-majority east remaining in India, causing a large-scale migration between the two. After the Punjabi Suba movement, Indian Punjab was reorganised on the basis of language in 1966, when its Haryanvi- and Hindi-speaking areas were carved out as Haryana, Pahari-speaking regions attached to Himachal Pradesh and the remaining, mostly Punjabi-speaking areas became the current state of Punjab. A separatist insurgency occurred in the state during the 1980s. At present, the economy of Punjab is the 15th-largest state economy in India with ₹5.29 trillion (US$66 billion) in gross domestic product and a per capita GDP of ₹151,367 (US$1,900), ranking 17th among Indian states. Since independence, Punjab is predominantly an agrarian society. It is the ninth-highest ranking among Indian states in human development index. Punjab has bustling tourism, music, culinary, and film industries.\\n\\n\\n== Etymology ==\\n\\n\\n== History ==\\n\\n\\n=== Ancient period ===\\nThe Punjab region is noted as the site of one of the earliest urban societies, the Indus Valley Civilization that flourished from about 3000 B.C. and declined rapidly 1,000 years later, follo', metadata={'title': 'Punjab, India', 'summary': \"Punjab ( ; Punjabi: [pənˈdʒɑːb]), historically known as Panchanada or Pentapotamia, is a state in northern India. Forming part of the larger Punjab region of the Indian subcontinent, the state is bordered by the Indian states of Himachal Pradesh to the north and northeast, Haryana to the south and southeast, and Rajasthan to the southwest; by the Indian union territories of Chandigarh to the east and Jammu and Kashmir to the north. It shares an international border with Punjab, a province of Pakistan to the west. The state covers an area of 50,362 square kilometres (19,445 square miles), which is 1.53% of India's total geographical area, making it the 19th-largest Indian state by area out of 28 Indian states (20th largest, if Union Territories are considered). With over 27 million inhabitants, Punjab is the 16th-largest Indian state by population, comprising 23 districts. Punjabi, written in the Gurmukhi script, is the most widely spoken and the official language of the state. The main ethnic group are the Punjabis, with Sikhs (57.7%) and Hindus (38.5%) forming the dominant religious groups. The state capital, Chandigarh, is a union territory and also the capital of the neighbouring state of Haryana. Three tributaries of the Indus River — the Sutlej, Beas, and Ravi — flow through Punjab.The history of Punjab has witnessed the migration and settlement of different tribes of people with different cultures and ideas, forming a civilisational melting pot. The ancient Indus Valley civilisation flourished in the region until its decline around 1900 BCE. Punjab was enriched during the height of the Vedic period, but declined in predominance with the rise of the Mahajanapadas. The region formed the frontier of initial empires during antiquity including Alexander's and the Maurya empires. It was subsequently conquered by the Kushan Empire, Gupta Empire, and then Harsha's Empire. Punjab continued to be settled by nomadic people; including the Huna, Turkic and the Mongols. Punjab came under Muslim rule c.\\u20091000 CE, and was part of the Delhi Sultanate and the Mughal Empire. Sikhism, based on the teachings of Sikh Gurus, emerged between the 15th and 17th centuries. Conflicts between the Mughals and the later Sikh Gurus precipitated a militarisation of the Sikhs, resulting in the formation of a confederacy after the weakening of the Mughal Empire, which competed for control with the larger Durrani Empire. This confederacy was united in 1801 by Maharaja Ranjit Singh, forming the Sikh Empire.The larger Punjab region was annexed by the British East India Company from the Sikh Empire in 1849. At the time of the independence of India from British rule in 1947, the Punjab province was partitioned along religious lines amidst widespread violence, with the Muslim-majority western portion becoming part of Pakistan and the Hindu- and Sikh-majority east remaining in India, causing a large-scale migration between the two. After the Punjabi Suba movement, Indian Punjab was reorganised on the basis of language in 1966, when its Haryanvi- and Hindi-speaking areas were carved out as Haryana, Pahari-speaking regions attached to Himachal Pradesh and the remaining, mostly Punjabi-speaking areas became the current state of Punjab. A separatist insurgency occurred in the state during the 1980s. At present, the economy of Punjab is the 15th-largest state economy in India with ₹5.29 trillion (US$66 billion) in gross domestic product and a per capita GDP of ₹151,367 (US$1,900), ranking 17th among Indian states. Since independence, Punjab is predominantly an agrarian society. It is the ninth-highest ranking among Indian states in human development index. Punjab has bustling tourism, music, culinary, and film industries.\", 'source': 'https://en.wikipedia.org/wiki/Punjab,_India'})]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to construct a graph based on the retrieved documents. For this purpose, we have implemented an `LLMGraphTransformermodule` that significantly simplifies constructing and storing a knowledge graph in a graph database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = ChatOpenAI(temperature=0,model_name = \"gpt-3.5-turbo-0125\")\n",
    "llm_transformer = LLMGraphTransformer(llm=llm)\n",
    "\n",
    "graph_documents = llm_transformer.convert_to_graph_documents(documents)\n",
    "graph.add_graph_documents(graph_documents, baseEntityLabel=True, include_source=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **LLM Selection and Support:**\n",
    "  - You can define which LLM you want the knowledge graph generation chain to use.\n",
    "  - Currently, we support only function calling models from OpenAI and Mistral.\n",
    "  - Expansion Plans: We plan to expand the LLM selection in the future.\n",
    "  - Example Model: In this example, we are using the latest GPT-4.\n",
    "  - Quality Dependency: Note that the quality of generated graph significantly depends on the model you are using.\n",
    "  - Optimal Model Usage: In theory, you always want to use the most capable one.\n",
    "- **Graph Generation Process:**\n",
    "  - The LLM graph transformers return graph documents.\n",
    "  - Integration with Neo4j: These documents can be imported to Neo4j via the add_graph_documents method.\n",
    "  - BaseEntityLabel Parameter: It assigns an additional __Entity__ label to each node, enhancing indexing and query performance.\n",
    "  - Include Source Parameter: It links nodes to their originating documents, facilitating data traceability and context understanding.\n",
    "- **Graph Inspection:**\n",
    "  - Visualization Tool: You can inspect the generated graph with yFiles visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b2b143c693874ee6ab0c121c4001beb5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "GraphWidget(layout=Layout(height='800px', width='100%'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ditectly show the graph resulting from  the given cypher query\n",
    "default_cypher = \"MATCH (s)-[r:!MENTIONS]->(t) RETURN s,r,t LIMIT 50\"\n",
    "\n",
    "def showGraph(cypher:str = default_cypher):\n",
    "    # create a neo4j session to run queries\n",
    "    driver = GraphDatabase.driver(\n",
    "        uri = os.environ[\"NEO4J_URI\"],\n",
    "        auth = (os.environ[\"NEO4J_USERNAME\"],os.environ[\"NEO4J_PASSWORD\"]))\n",
    "    session = driver.session()\n",
    "    widget = GraphWidget(graph = session.run(cypher).graph())\n",
    "    widget.node_label_mapping = \"id\"\n",
    "    # display(widget)\n",
    "    return widget\n",
    "\n",
    "showGraph()                               \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.Hybrid Retrieval for RAG:\n",
    "  - After graph generation, a hybrid retrieval approach is utilized for RAG (Retrieval-Augmented Generation) \n",
    "  applications.  \n",
    "![](https://miro.medium.com/v2/resize:fit:700/1*TJJBOZN9auUioEnqQo-Qdw.png)\n",
    "  - **Retrieval Process:**\n",
    "    - User Interaction: The process begins with a user posing a question.\n",
    "    - RAG Retriever: The question is directed to an RAG retriever.\n",
    "    - Retrieval Techniques: This retriever employs keyword and vector searches to sift through unstructured text data.\n",
    "    - Graph Integration: It combines retrieved information with data from the knowledge graph.\n",
    "    - Single Database System: Since Neo4j supports both keyword and vector indexes, all three retrieval options can be implemented using a single database system.\n",
    "    - Final Answer Generation: The collected data from these sources is fed into an LLM to generate and deliver the final answer.\n",
    "- **Unstructured Data Retriever:**\n",
    "  - **Neo4jVector.from_existing_graph Method:**\n",
    "    - Functionality: You can utilize this method to add both keyword and vector retrieval to documents.\n",
    "    - Configuration: It configures keyword and vector search indexes for a hybrid search approach.\n",
    "    - Target Nodes: This method targets nodes labeled Document.\n",
    "    - Additional Functionality: It calculates text embedding values if they are missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_index = Neo4jVector.from_existing_graph(\n",
    "    OpenAIEmbeddings(),\n",
    "    search_type=\"hybrid\",\n",
    "    node_label=\"Document\",\n",
    "    text_node_properties=[\"text\"],\n",
    "    embedding_node_property=\"embedding\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Vector Index Utilization:**\n",
    "  - **Similarity_Search Method:**\n",
    "    - Usage: The vector index can be called with the similarity_search method.\n",
    "\n",
    "# 8.Graph Retriever:\n",
    "  - **Configuration and Process:**\n",
    "    - Involvement and Flexibility: Configuring a graph retrieval is more involved but offers more freedom.\n",
    "    - Methodology: It utilizes a full-text index to identify relevant nodes and then returns their direct neighborhood.\n",
    "  - **Example and Visualization:**\n",
    "    - Example Diagram: \n",
    "    \n",
    "  ![graph](https://raw.githubusercontent.com/tomasonjo/blogs/master/neighbor.png)\n",
    "  - **Retriever Workflow:**\n",
    "    - Identification of Relevant Entities: The graph retriever starts by identifying relevant entities in the input.\n",
    "    - Structured Output Instruction: For simplicity, the LLM is instructed to identify people, organizations, and locations.\n",
    "  - **Utilization of LCEL:**\n",
    "    - Method: To achieve this, LCEL is used with the newly added `with_structured_output` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\ibm26\\anaconda3\\envs\\dev-llm\\lib\\site-packages\\langchain_core\\_api\\beta_decorator.py:86: LangChainBetaWarning: The function `with_structured_output` is in beta. It is actively being worked on, so the API may change.\n",
      "  warn_beta(\n"
     ]
    }
   ],
   "source": [
    "# Retriever\n",
    "graph.query(\n",
    "    \"CREATE FULLTEXT INDEX entity IF NOT EXISTS FOR (e:__Entity__) ON EACH [e.id]\")\n",
    "\n",
    "# Extract entities from text\n",
    "class Entities(BaseModel):\n",
    "    \"\"\"Identifying information about entities.\"\"\"\n",
    "\n",
    "    names: List[str] = Field(\n",
    "        ...,\n",
    "        description=\"All the person, organization, or business entities that \"\n",
    "        \"appear in the text\",\n",
    "    )\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", \"You are extracting organization and person entities from the text.\",),\n",
    "        (\"human\",\"Use the given format to extract information from the following input: {question}\",),\n",
    "    ]\n",
    ")\n",
    "\n",
    "entity_chain = prompt | llm.with_structured_output(Entities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['India', 'south asia']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entity_chain.invoke({\"question\": \"India is located in south asia?\"}).names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.Full Knowledge Graph\n",
    "\n",
    "Great, now that we can detect entities in the question, let's use a full-text index to map them to the knowledge graph. First, we need to define a full-text index and a function that will generate full-text queries that allow a bit of misspelling, which we won't go into much detail here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_full_text_query(input: str) -> str:\n",
    "    \"\"\"\n",
    "    Generate a full-text search query for a given input string.\n",
    "\n",
    "    This function constructs a query string suitable for a full-text search.\n",
    "    It processes the input string by splitting it into words and appending a\n",
    "    similarity threshold (~2 changed characters) to each word, then combines\n",
    "    them using the AND operator. Useful for mapping entities from user questions\n",
    "    to database values, and allows for some misspelings.\n",
    "    \"\"\"\n",
    "    full_text_query = \"\"\n",
    "    words = [el for el in remove_lucene_chars(input).split() if el]\n",
    "    for word in words[:-1]:\n",
    "        full_text_query += f\" {word}~2 AND\"\n",
    "    full_text_query += f\" {words[-1]}~2\"\n",
    "    return full_text_query.strip()\n",
    "\n",
    "# Fulltext index query\n",
    "def structured_retriever(question: str) -> str:\n",
    "    \"\"\"\n",
    "    Collects the neighborhood of entities mentioned\n",
    "    in the question\n",
    "    \"\"\"\n",
    "    result = \"\"\n",
    "    entities = entity_chain.invoke({\"question\": question})\n",
    "    for entity in entities.names:\n",
    "        response = graph.query(\n",
    "            \"\"\"CALL db.index.fulltext.queryNodes('entity', $query, {limit:2})\n",
    "            YIELD node,score\n",
    "            CALL {\n",
    "              MATCH (node)-[r:!MENTIONS]->(neighbor)\n",
    "              RETURN node.id + ' - ' + type(r) + ' -> ' + neighbor.id AS output\n",
    "              UNION\n",
    "              MATCH (node)<-[r:!MENTIONS]-(neighbor)\n",
    "              RETURN neighbor.id + ' - ' + type(r) + ' -> ' +  node.id AS output\n",
    "            }\n",
    "            RETURN output LIMIT 50\n",
    "            \"\"\",\n",
    "            {\"query\": generate_full_text_query(entity)},\n",
    "        )\n",
    "        result += \"\\n\".join([el['output'] for el in response])\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Large Language Model - TASK -> General-Purpose Language Generation\n",
      "Large Language Model - TASK -> Natural Language Processing\n",
      "Large Language Model - TASK -> Classification\n",
      "Large Language Model - MODEL -> Artificial Neural Networks\n",
      "Large Language Model - MODEL -> Decoder-Only Transformer-Based Architecture\n",
      "Large Language Model - MODEL -> Recurrent Neural Network Variants\n",
      "Large Language Model - MODEL -> Mamba\n",
      "Large Language Model - METHOD -> Fine Tuning\n",
      "Large Language Model - MODEL -> Gpt-3\n",
      "Large Language Model - COMPONENT -> Syntax\n",
      "Large Language Model - COMPONENT -> Semantics\n",
      "Large Language Model - COMPONENT -> Ontology\n",
      "Large Language Model - COMPONENT -> Inaccuracies\n",
      "Large Language Model - COMPONENT -> Biases\n",
      "Large Language Model - COMPANY -> Openai\n",
      "Large Language Model - MODEL -> Chatgpt\n",
      "Large Language Model - MODEL -> Microsoft Copilot\n",
      "Large Language Model - COMPANY -> Google\n",
      "Large Language Model - MODEL -> Palm\n",
      "Large Language Model - MODEL -> Gemini\n",
      "Neurips Conference - CONFERENCE -> Google Researchers\n",
      "Google Researchers - ARCHITECTURE -> Transformer Architecture\n",
      "Transformer Architecture - COMPONENT -> Attention Mechanism\n",
      "Attention Mechanism - RESEARCHER -> Bahdanau\n",
      "Bahdanau - TECHNOLOGY -> Seq2Seq Technology\n",
      "Seq2Seq Technology - MODEL -> Bert\n",
      "Large Language Model - MODEL -> Gpt-1\n",
      "Large Language Model - MODEL -> Gpt-2\n",
      "Gpt-3 - TECHNOLOGY -> Api\n",
      "Chatgpt - RELEASED_IN -> 2022\n",
      "Chatgpt - DEPLOYED_ON -> Browser-Based\n",
      "Chatgpt - TARGETED_TOWARDS -> General Population\n",
      "Chatgpt - CAUSED -> Media Hype\n",
      "Gpt-4 - RELEASED_IN -> 2023\n",
      "Gpt-4 - PRAISED_FOR -> Increased Accuracy\n",
      "Gpt-4 - PRAISED_FOR -> Multimodal Capabilities\n",
      "Gpt-4 - DESCRIBED_AS -> Holy Grail\n",
      "Openai - WITHHELD -> Did Not Reveal Architecture\n",
      "Openai - WITHHELD -> Did Not Reveal Number Of Parameters\n",
      "Gpt-4 - MENTIONED_WITH -> Competing Language Models\n",
      "Gpt-4 - OBSERVED_IN -> Playing Catch-Up\n",
      "Google - DEVELOPED -> T5-11B\n",
      "Google - DEVELOPED -> Palm-E\n",
      "Google - DEVELOPED -> Bard (Gemini Pro)\n",
      "T5-11B - RELEASED_IN -> 2019\n",
      "Palm-E - RELEASED_IN -> 2022\n",
      "Claude 3 - RELEASED_IN -> 2024\n",
      "Bard (Gemini Pro) - SURPASSED -> January 26, 2024\n",
      "Bard (Gemini Pro) - SURPASSED -> Gpt-4\n",
      "Gpt-4 - NOT_SURPASSED -> Gpt-4-TurboLarge Language Model - TASK -> General-Purpose Language Generation\n",
      "Large Language Model - TASK -> Natural Language Processing\n",
      "Large Language Model - TASK -> Classification\n",
      "Large Language Model - MODEL -> Artificial Neural Networks\n",
      "Large Language Model - MODEL -> Decoder-Only Transformer-Based Architecture\n",
      "Large Language Model - MODEL -> Recurrent Neural Network Variants\n",
      "Large Language Model - MODEL -> Mamba\n",
      "Large Language Model - METHOD -> Fine Tuning\n",
      "Large Language Model - MODEL -> Gpt-3\n",
      "Large Language Model - COMPONENT -> Syntax\n",
      "Large Language Model - COMPONENT -> Semantics\n",
      "Large Language Model - COMPONENT -> Ontology\n",
      "Large Language Model - COMPONENT -> Inaccuracies\n",
      "Large Language Model - COMPONENT -> Biases\n",
      "Large Language Model - COMPANY -> Openai\n",
      "Large Language Model - MODEL -> Chatgpt\n",
      "Large Language Model - MODEL -> Microsoft Copilot\n",
      "Large Language Model - COMPANY -> Google\n",
      "Large Language Model - MODEL -> Palm\n",
      "Large Language Model - MODEL -> Gemini\n",
      "Neurips Conference - CONFERENCE -> Google Researchers\n",
      "Google Researchers - ARCHITECTURE -> Transformer Architecture\n",
      "Transformer Architecture - COMPONENT -> Attention Mechanism\n",
      "Attention Mechanism - RESEARCHER -> Bahdanau\n",
      "Bahdanau - TECHNOLOGY -> Seq2Seq Technology\n",
      "Seq2Seq Technology - MODEL -> Bert\n",
      "Large Language Model - MODEL -> Gpt-1\n",
      "Large Language Model - MODEL -> Gpt-2\n",
      "Gpt-3 - TECHNOLOGY -> Api\n",
      "Chatgpt - RELEASED_IN -> 2022\n",
      "Chatgpt - DEPLOYED_ON -> Browser-Based\n",
      "Chatgpt - TARGETED_TOWARDS -> General Population\n",
      "Chatgpt - CAUSED -> Media Hype\n",
      "Gpt-4 - RELEASED_IN -> 2023\n",
      "Gpt-4 - PRAISED_FOR -> Increased Accuracy\n",
      "Gpt-4 - PRAISED_FOR -> Multimodal Capabilities\n",
      "Gpt-4 - DESCRIBED_AS -> Holy Grail\n",
      "Openai - WITHHELD -> Did Not Reveal Architecture\n",
      "Openai - WITHHELD -> Did Not Reveal Number Of Parameters\n",
      "Gpt-4 - MENTIONED_WITH -> Competing Language Models\n",
      "Gpt-4 - OBSERVED_IN -> Playing Catch-Up\n",
      "Google - DEVELOPED -> T5-11B\n",
      "Google - DEVELOPED -> Palm-E\n",
      "Google - DEVELOPED -> Bard (Gemini Pro)\n",
      "T5-11B - RELEASED_IN -> 2019\n",
      "Palm-E - RELEASED_IN -> 2022\n",
      "Claude 3 - RELEASED_IN -> 2024\n",
      "Bard (Gemini Pro) - SURPASSED -> January 26, 2024\n",
      "Bard (Gemini Pro) - SURPASSED -> Gpt-4\n",
      "Gpt-4 - NOT_SURPASSED -> Gpt-4-Turbo\n"
     ]
    }
   ],
   "source": [
    "print(structured_retriever(\"India is located in south asia?\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10.Final retriever\n",
    "As we mentioned at the start, we'll combine the unstructured and graph retriever to create the final context that will be passed to an LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retriever(question: str):\n",
    "    print(f\"Search query: {question}\")\n",
    "    structured_data = structured_retriever(question)\n",
    "    unstructured_data = [el.page_content for el in vector_index.similarity_search(question)]\n",
    "    final_data = f\"\"\"Structured data:\n",
    "    {structured_data}\n",
    "    Unstructured data:\n",
    "    {\"#Document \". join(unstructured_data)}\n",
    "    \"\"\"\n",
    "    return final_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we are dealing with Python, we can simply concatenate the outputs using the f-string.\n",
    "# 11.Defining the RAG chain\n",
    "We have successfully implemented the retrieval component of the RAG. First, we will introduce the query rewriting part that allows conversational follow up questions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Condense a chat history and follow-up question into a standalone question\n",
    "_template = \"\"\"Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question,\n",
    "in its original language.\n",
    "Chat History:\n",
    "{chat_history}\n",
    "Follow Up Input: {question}\n",
    "Standalone question:\"\"\"  # noqa: E501\n",
    "CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(_template)\n",
    "\n",
    "def _format_chat_history(chat_history: List[Tuple[str, str]]) -> List:\n",
    "    buffer = []\n",
    "    for human, ai in chat_history:\n",
    "        buffer.append(HumanMessage(content=human))\n",
    "        buffer.append(AIMessage(content=ai))\n",
    "    return buffer\n",
    "\n",
    "_search_query = RunnableBranch(\n",
    "    # If input includes chat_history, we condense it with the follow-up question\n",
    "    (\n",
    "        RunnableLambda(lambda x: bool(x.get(\"chat_history\"))).with_config(\n",
    "            run_name=\"HasChatHistoryCheck\"\n",
    "        ),  # Condense follow-up question and chat into a standalone_question\n",
    "        RunnablePassthrough.assign(\n",
    "            chat_history=lambda x: _format_chat_history(x[\"chat_history\"])\n",
    "        )\n",
    "        | CONDENSE_QUESTION_PROMPT\n",
    "        | ChatOpenAI(temperature=0)\n",
    "        | StrOutputParser(),\n",
    "    ),\n",
    "    # Else, we have no chat history, so just pass through the question\n",
    "    RunnableLambda(lambda x : x[\"question\"]),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "template = \"\"\"Answer the question based only on the following context:\n",
    "{context}\n",
    "\n",
    "Question: {question}\n",
    "Use natural language and be concise.\n",
    "Answer:\"\"\"\n",
    "prompt = ChatPromptTemplate.from_template(template)\n",
    "\n",
    "chain = (\n",
    "    RunnableParallel(\n",
    "        {\n",
    "            \"context\": _search_query | retriever,\n",
    "            \"question\": RunnablePassthrough(),\n",
    "        }\n",
    "    )\n",
    "    | prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search query: What is large language model?\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'A large language model is an artificial neural network used for general-purpose language generation, natural language processing, and classification tasks. It learns statistical relationships from text documents during training to generate text and understand syntax, semantics, ontology, inaccuracies, and biases in human language.'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": \"What is large language model?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search query: What is the name of the first large language model?\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The name of the first large language model is GPT-1.'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke(\n",
    "    {\n",
    "        \"question\": \"What is the name of first large language model?\",\n",
    "        \"chat_history\": [(\"What is large language model?\", \"A large language model is an artificial neural network used for general-purpose language generation, natural language processing, and classification tasks. It learns statistical relationships from text documents during training to generate text and understand syntax, semantics, ontology, inaccuracies, and biases in human language.\")],\n",
    "    }\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dev-llm",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}