How do I create a custom prompt for create_pandas_dataframe_agent ?

[monkeydust]

I want to add my own template to this create_pandas_dataframe_agent to provide human context on the dataframe it is using.

How can do that?

Txs

[dosubot[bot]]

🤖

Hey @monkeydust!

I'm Dosu, an AI bot here to assist you with your queries and issues related to the LangChain repository. I'm more than happy to help you while we wait for a human maintainer. Let's get started on solving your issue, shall we?

To add a custom template to the create_pandas_dataframe_agent in LangChain, you can provide your custom template as the prefix and suffix parameters when calling the function. The prefix and suffix are used to construct the prompt that is sent to the language model.

Here is an example of how you can do this:

from langchain_experimental.agents.agent_toolkits.pandas.base import create_pandas_dataframe_agent
from langchain.agents.types import AgentType
from langchain.schema.language_model import BaseLanguageModel
import pandas as pd

# Assuming you have a language model instance
llm = BaseLanguageModel()

# Create a pandas DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Define your custom prefix and suffix
custom_prefix = "This is my custom prefix. "
custom_suffix = " This is my custom suffix."

# Create the pandas dataframe agent
agent = create_pandas_dataframe_agent(
    llm=llm,
    df=df,
    agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    prefix=custom_prefix,
    suffix=custom_suffix,
    include_df_in_prompt=True,
    number_of_head_rows=5
)

In this example, the custom_prefix and custom_suffix are the custom templates that you want to add. The prefix is added at the beginning of the prompt and the suffix is added at the end. The include_df_in_prompt parameter is set to True to include the DataFrame in the prompt, and number_of_head_rows is set to 5 to include the first 5 rows of the DataFrame in the prompt.

You can find more details about this in the LangChain codebase at libs/experimental/langchain_experimental/agents/agent_toolkits/pandas/base.py.

If you want to create an agent in LangChain that accepts both a "dataframe" and "user_input" as input variables, you can modify the create_prompt method in the StructuredChatAgent class to include "dataframe" and "user_input" in the input_variables list. You can find more details about this in the solved issue here.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• Does anyone created agent with tool which takes "dataframe" and "user_input" as input variables in langchain?
• libs/experimental/langchain_experimental/agents/agent_toolkits/pandas/base.py
• libs/langchain/langchain/agents/agent_toolkits/pandas/init.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[muksgupta]

@dosu-bot
I am getting below error
Invalid prompt schema; check for mismatched or missing input parameters. {'df', 'input', 'agent_scratchpad'} (type=value_error)

[muksgupta]

@dosu-bot
I am getting below error
Invalid prompt schema; check for mismatched or missing input parameters. {'df', 'input', 'agent_scratchpad'} (type=value_error)

[barauna-lo]

I'm building an AI assistant where the non-data science user needs to ask a question to a dataset.

The way I found it was to first do a FewShotPromptTemplate with some examples of translating the question at a higher level for a data science task. Then passing the question to create_pandas_dataframe_agent and thus extracting the suggested information.

It turns out that create_pandas_dataframe_agent doesn't return the same answer to the same question, and doesn't even get the queries right every time (even for temperature = 0).

I would like to create a FewShotPromptTemplate within the agent itself and pass some pandas code exemple like we do in FewShotPromptTemplate. How to do this?

Note: I tried using the prefix langchain_experimental.agents.agent_toolkits.pandas.prompt import PREFIX but the results are not the best.

[zakatlassi]

Did you manage to solve this ?

[barauna-lo]

Not yet. I tried to use the prefix in the agent but doesn't work very well. Do you have any ideia how to solve it?

[zakatlassi]

I'm also trying to use the prefix. I've managed to reduce the numbers of Output Parsing errors, but for the inconsistency of the answers I still struggle.

[maximeperrindev]

Hi all,

Just to let you know the used prompts for create_pandas_dataframe_agent are :

PREFIX = """Answer the following questions as best you can. You have access to the following tools:"""
FORMAT_INSTRUCTIONS = """Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question"""
SUFFIX = """Begin!

Question: {input}
Thought:{agent_scratchpad}"""

As you said, you can only pass custom instructions through the prefix or suffix. Everything is concatenated together so it shouldn't be a problem. The best way to add humain context is to pass it through the suffix as it will be the first instruction.