PydanticOutputParser failing with gpt-4o - works with gpt-4-turbo

[danielbentes]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import os
from dotenv import load_dotenv
from typing import List, Optional
from pydantic.v1 import BaseModel, Field, ValidationError
from langchain_core.prompts import ChatPromptTemplate
from langchain.output_parsers import PydanticOutputParser
from langchain_core.runnables import RunnableSequence
from langchain_openai import ChatOpenAI

class SuggestiveQuestions(BaseModel):
    questions: List[str] = Field(..., description="List of suggestive questions a potential buyer might ask a real estate broker.")
    
class SuggestivesChain:
    def __init__(self, model):
        self.model = model
    
    def build_chain(self):
        suggestives_extraction_template = """
        # Instructions:
        Generate a list of 5 targeted follow-up questions a property buyer might ask a real estate broker, aiming to refine and detail their specific needs and preferences regarding property type, size, location, budget, and key features. These questions should be crafted to help the broker gather necessary details to effectively search for and showcase properties that meet the buyer's criteria.

        # Original Question Examples:
        "I'm interested in purchasing a home in Spain, specifically in Marbella."
        "Do you have a property close to the beach with at least 4 bedrooms for less than 2.5 million euros?"
        
        # Follow-Up Question Examples:
        ## For the first example:
        "What types of properties are available within my budget in different parts of Marbella?"
        "Can you identify properties in Marbella that offer specific features such as a sea view, private pool, or proximity to local amenities like schools and shopping centers?"
        "Can you identify properties in Marbella that offer specific features such as private swimming pools, large gardens, and proximity to top-rated schools?"
        "Are there upcoming residential developments or areas in Marbella that you would recommend based on my interest in modern, energy-efficient family homes?"
        "Can you highlight properties in Marbella that are particularly family-friendly, with emphasis on safety, schools, and community amenities?"
        
        ## For the second example:
        "Are there similar properties with at least 4 bedrooms in nearby areas that might offer a lower price point?"
        "Can you identify any properties that offer unique features such as private pools, large gardens, or exceptional views of the sea within my specified budget and desired location?"
        "Could you elaborate on the neighborhood amenities, such as schools, shopping centers, and public transportation options, available near the beachfront properties in Fuengirola?"
        "Can you provide details on properties that are within a 10-minute walk from the beach, featuring at least 4 bedrooms and 3 bathrooms, and priced under 2.5 million euros?"
        "Are there any properties close to the beach that not only meet the 4-bedroom criterion but also include additional amenities such as a swimming pool, garden, or garage?"
        
        # Original question:
        {question}

        # Output Format:
        Wrap the output in `json` tags\n{format_instructions}
        """
        suggestives_prompt = ChatPromptTemplate.from_template(suggestives_extraction_template)
        suggestives_parser = PydanticOutputParser(pydantic_object=SuggestiveQuestions)
        
        chain = suggestives_prompt.partial(format_instructions=suggestives_parser.get_format_instructions()) | self.model | suggestives_parser

        return chain

    def invoke(self, data):
        chain = RunnableSequence(self.build_chain())
        return chain.invoke({"question":data})
    
    def ainvoke(self, data):
        chain = RunnableSequence(self.build_chain())
        return chain.ainvoke({"question":data})

# Load environment variables
load_dotenv()

# Load OpenAI API key from .env file and set it globally
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
#model = ChatOpenAI(model="gpt-4-turbo", temperature=0, model_kwargs={"seed": 1337})
model = ChatOpenAI(model="gpt-4o", temperature=0, model_kwargs={"seed": 1337})

# Generata preferences for the user based on their interactions
chain = SuggestivesChain(model)

questions = chain.invoke("What types of properties are available within my budget in different parts of Marbella?")
print(questions)

Description

Using gpt-4-turbo I get the correct response:
https://smith.langchain.com/public/e8a3576d-4262-48fe-b025-bf4a98848c9b/r

Using gpt-4o I get a validation error:
OutputParserException('Failed to parse SuggestiveQuestions from completion {"properties": {"questions": {"title": "Questions", "description": "List of suggestive questions a potential buyer might ask a real estate broker.", "type": "array", "items": {"type": "string"}}}, "required": ["questions"]}. Got: 1 validation error for SuggestiveQuestions\nquestions\n field required (type=value_error.missing)')

Same code, different models.

System Info

langchain_version: "0.1.16"
library: "langsmith"
platform: "macOS-14.4.1-arm64-arm-64bit"
py_implementation: "CPython"
runtime: "python"
runtime_version: "3.11.8"
sdk: "langsmith-py"
sdk_version: "0.1.52"

langchain==0.1.16
langchain-community==0.0.36
langchain-core==0.1.48
langchain-openai==0.1.3
langchain-text-splitters==0.0.1
langchainhub==0.1.15

[dosubot[bot]]

To address the validation error with the PydanticOutputParser when using gpt-4o, it's crucial to ensure the output from gpt-4o matches the expected schema of your Pydantic model. The error indicates a mismatch between the expected and actual output structures.

1. Review and Adjust Your Pydantic Model: Confirm the expected JSON schema using the get_format_instructions method of the PydanticOutputParser. If necessary, adjust your Pydantic model to align with the output structure of gpt-4o.

2. Modify the Language Model Prompt: If the output structure still doesn't match, consider modifying the prompt to ensure the output from gpt-4o aligns with your Pydantic model's expectations.

3. Implement Post-Processing: As a last resort, add a step to transform the gpt-4o output into the correct structure before parsing it with your Pydantic model.

Ensure the output from gpt-4o is correctly formatted to match your Pydantic model's schema. This might involve adjusting the model or the output data structure before parsing. For more detailed guidance, refer to the LangChain documentation on PydanticOutputParser.

Details

Sources

• libs/core/langchain_core/output_parsers/pydantic.py
• libs/langchain/langchain/chains/structured_output/base.py
• libs/langchain/langchain/schema/output.py
• docs/docs/how_to/output_parser_structured.ipynb
• docs/docs/how_to/structured_output.ipynb
• libs/community/langchain_community/callbacks/openai_info.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[danielbentes]

Well... obviously there is a discrepancy between get_format_instructions method of the PydanticOutputParser and how the gpt-4o model behaves. Meaning that the instructions are not good/clear enough for the model. So it responds with a format that doesn't conform to what is expected.

A work around that works is to add this to the prompt template after the instructions (or just create my own format instructions and skip the PydanticOutputParser alltogether:

"Don't encapsulate the response with markdown. Don't output the schema itself."

[benjaminbascary97]

Im following this as I am having the same issue with the following:

agent = create_react_agent(llm, tools, prompt_template)

agent_executor = AgentExecutor(agent=agent, tools=tools, handle_parsing_errors=True)

It does not work passing a custom instruction asking not to return markdown.

[jefromyers]

I had the same issues using from pydantic.v1 import BaseModel, Field, ValidationError, but when using pydantic v2 it has been working.

[danielbentes]

Haven't tried with pydantic v2 since there are compatibility issues (https://python.langchain.com/v0.1/docs/guides/development/pydantic_compatibility/).

Will have to look at that.

[hinthornw]

This isn't pydantic v2 related this is the fact that the model generated somethign that didn't conform to the schema, because 4o is faster but behaves differently than other model variants. Would suggest catching the errors and repromtping after formatting as errors in a tool messages for now. We're looking into making a nicer re-prompting abstraction though for similar issues

[danielbentes]

Hi @hinthornw,

Thanks for your input. I agree that the issue with the PydanticOutputParser is due to the differences in behavior between the gpt-4o and other model variants, which results in outputs that do not conform to the expected schema.

To address this, I suggest two potential solutions:

1. Modify the Prompt Template: Update the formatting instruction template to be more explicit for the gpt-4o model. This would involve adding clearer instructions to avoid encapsulating the response with markdown or any other non-compliant format.

2. Harden Error Handling: Enhance the error handling to catch validation errors and re-prompt or reformat the response accordingly. This could involve implementing a more robust error-catching mechanism that provides feedback to the model and requests a correctly formatted response.

I believe these adjustments could help mitigate the discrepancies and ensure more consistent outputs from the gpt-4o model. Additionally, as you mentioned, implementing a more user-friendly re-prompting abstraction would greatly enhance the robustness of the system.

[proy9714]

Anyone found the fix yet?

[danielbentes]

The "fix" so far has been to try to tweak the prompt itself with additional instructions. You can't rely on the default instructions from the PydanticOutputParser. @dosu should notify the devs in case they have a better solution...