Is that possible to get my response via Query pipe line instead of 'Query_engine_query'?

[mraguth]

Apologies for the basic question as I am new to llama_index. Our research has shown the impressive capabilities of llama_index in the 'pipeline' for handling complex tasks. The potential of llama_index has truly surprised me.

However, my current query takes a different direction. Upon exploring the 'RecursiveRetriever in our RetrieverQueryEngine', I noticed its effectiveness in retrieval, particularly with structured data processing like vectoring and querying using query_engine_query is great.

My question now is whether it is feasible to implement a pipeline process similar to the one used in the 'Pandas query pipeline' within the RecursiveRetriever of our RetrieverQueryEngine?

I aim to transition from using query_engine.query, as demonstrated in the documentation, to a pipeline approach similar to this example:
response = qp.run(query_str="What is the correlation between survival and age?").

Your guidance on this matter would be greatly appreciated.

These are the references I am using for LlamaIndex
Recursive Retriever + Query Engine Demo https://docs.llamaindex.ai/en/stable/examples/query_engine/pdf_tables/recursive_retriever/

Pandas query pipe line: https://docs.llamaindex.ai/en/stable/examples/pipeline/query_pipeline_pandas/?h=pandas+query

[dosubot[bot]]

To implement a pipeline querying process within the RecursiveRetriever of your RetrieverQueryEngine, akin to the Pandas query pipeline, you can enhance the RecursiveRetrieverSmallToBigPack class with a query_pipeline method. This method will orchestrate the execution of a series of query operations, allowing for a flexible and modular approach to complex queries. Here's how you can set it up:

class RecursiveRetrieverSmallToBigPack(BaseLlamaPack):
    def query_pipeline(self, initial_query: str, operations: List[Callable[[Any], Any]]) -> Any:
        """
        Execute a pipeline of query operations on the result of an initial query.

        :param initial_query: The initial query string to be executed.
        :param operations: A list of functions that take a query result and return a modified result.
        :return: The final result after all operations have been applied.
        """
        # Execute the initial query
        result = self.query_engine.query(initial_query)

        # Sequentially apply each operation in the pipeline to the result
        for operation in operations:
            result = operation(result)

        return result

In this setup, query_pipeline first runs an initial query through the query_engine. It then iterates over a list of operations, each of which is a function that processes the query result in some way, and applies these operations in sequence. This design allows for the construction of complex query pipelines by chaining together individual query operations, offering a high degree of flexibility and modularity.

To use this method, you would define your query operations as functions or lambdas that can be passed to query_pipeline along with your initial query. This approach enables you to build and customize your query processing pipeline to fit your specific needs, leveraging the powerful querying capabilities of the RecursiveRetriever in a dynamic and efficient manner.

Details

Sources

• llama-index-packs/llama-index-packs-recursive-retriever/llama_index/packs/recursive_retriever/small_to_big/base.py
• llama-index-legacy/tests/query_pipeline/test_query.py
• llama-index-core/tests/query_pipeline/test_query.py
• llama-index-core/llama_index/core/retrievers/recursive_retriever.py
• llama-index-legacy/llama_index/legacy/retrievers/recursive_retriever.py
• llama-index-legacy/llama_index/legacy/query_engine/retriever_query_engine.py
• llama-index-core/llama_index/core/query_engine/retriever_query_engine.py
• docs/docs/examples/pipeline/query_pipeline_pandas.ipynb
• docs/docs/examples/query_engine/pandas_query_engine.ipynb
• llama-index-experimental/llama_index/experimental/query_engine/pandas/pandas_query_engine.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.