Support with_structured_output in ChatAnthropicVertex

[shotarok]

ChatAnthropicVertex doesn't support with_structured_output although ChatVertexAI support it.

[lkuligin]

would you like to work on this one?

[shotarok]

Yes, I'm happy to contribute to this project! I plan to use PydanticOutputParser and JsonOutputParser because ChatAnthropicVertex doesn't support function_calling.

[lkuligin]

we should probably start with adding function calling support first.

[shotarok]

Oh, I had assumed Claude API on Vertex AI doesn't support function calling since the doc doesn't mention it, but I'll test a function call. I agree. If it works, supporting function call would be the first task.

[shotarok]

I'm afraid I couldn't confirm that Claude on Vertex AI supports function calling. I got this error: "tools: Extra inputs are not permitted" with claude-3-haiku@20240307 on Vertex AI. So, I'll create a PR with PydanticOutputParser and JsonOutputParser . I attached the succeeded and failed curl commands, just in case.

A succeeded curl without function calling

$ curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "anthropic_version": "vertex-2023-10-16",
    "max_tokens": 100,
    "messages": [{"role": "user", "content": "What is the weather like in San Francisco?"}]
}' \
"https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/anthropic/models/${MODEL}:rawPredict"
> {"id":"msg_vrtx_01RYFRNxnQZRDsp71Jq1aKXH","type":"message","role":"assistant","model":"claude-3-haiku-20240307","stop_sequence":null,"usage":{"input_tokens":16,"output_tokens":100},"content":[{"type":"text","text":"The weather in San Francisco can be quite variable, but here's a general overview:\n\n- Mild Temperatures - San Francisco has a Mediterranean climate, with average highs around 65°F (18°C) year-round. Summers are cool, with highs in the 60s or 70s Fahrenheit (15-25°C). Winters are mild, with highs in the 50s and 60s Fahrenheit (10-20"}],"stop_reason":"max_tokens"}

A failed curl with function calling

$ curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "anthropic_version": "vertex-2023-10-16",
    "max_tokens": 100,
    "tools": [{
        "name": "get_weather",
        "description": "Get the current weather in a given location",
        "input_schema": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                },
                "unit": {
                    "type": "string",
                    "enum": ["celsius", "fahrenheit"],
                    "description": "The unit of temperature, either \"celsius\" or \"fahrenheit\""
                }
            },
            "required": ["location"]
        }
    }],"messages": [{"role": "user", "content": "What is the weather like in San Francisco?"}]
}' \
"https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/anthropic/models/${MODEL}:rawPredict"
> {"type":"error","error":{"type":"invalid_request_error","message":"tools: Extra inputs are not permitted"}}

[shotarok]

After reading langchain-core, I understand the other chat models using PydanticOutputParser and JsonOutputParser assume the LLM API can control output format as JSON like the JSON mode in OpenAI.

Until Claude on Vertex AI supports tools, we may be unable to support with_structured_output in ChatAnthropicVertex. Instead, we can use StructuredOutputParser with giving a prompt by ourselves 1.

Please let me know if you have a different implementation idea. Thanks!

[lkuligin]

yes, until tools are supported we should rely on a model following the prompt.
you can even try to add an optional extra step to the chain to convert the output to a desired format like here:

langchain-google/libs/vertexai/langchain_google_vertexai/chains.py

Line 92 in 3b34f65

use_extra_step: bool = False,