Problems in accessing Claude 3 models with Vertex Model Garden

[Vincsi]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# Model Garden
from langchain_google_vertexai import VertexAIModelGarden
LOCATION="us-central1" # or "europe-west4"
llm = VertexAIModelGarden(project=LOCATION, endpoint_id="YOUR ENDPOINT_ID")

# VertexAI
from langchain_google_vertexai import VertexAI
model = VertexAI(model_name="claude-3-haiku@20240307")
# ERROR: NotFound: 404 Publisher Model `publishers/google/models/claude-3-haiku@20240307` is not found.

Description

Anthropic models in GCP model garden are directly managed by google and do not have an endpoint like other LLM (e.g. deployed version of LLAMA 2).
How can Anthropic models be accessed using the Model Garden?

I also tried using the VertexAI integration by listing the Claude3 model instead of Gemini but I receive the following error

• NotFound: 404 Publisher Model **publishers/google/models/claude-3-haiku@20240307** is not found.

When I explore the Claude 3 model card on GCP the Anthropic models are stored in a different location:

• 'publishers/anthropic/models/claude-3-haiku@20240307'

System Info

System Information

OS: Linux
OS Version: #1 SMP Debian 4.19.304-1 (2024-01-09)
Python Version: 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]

Package Information

langchain_core: 0.1.31
langchain: 0.1.12
langchain_community: 0.0.28
langsmith: 0.1.24
langchain_experimental: 0.0.50
langchain_google_genai: 0.0.1
langchain_google_vertexai: 0.1.0
langchain_openai: 0.0.8
langchain_text_splitters: 0.0.1
langchainhub: 0.1.14
langgraph: 0.0.19

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

[dosubot[bot]]

Hey @Vincsi, great to see you back! Hope you've been well since our last chat.

Based on the information you've provided, it seems like you're trying to access the Claude 3 model from the Anthropic publisher, but the model location you're using is for the Google publisher. This is likely the cause of the NotFound error you're encountering.

In your code, you're using the VertexAI class to access the model with the following line:

model = VertexAI(model_name="claude-3-haiku@20240307")

However, the model name you're using ("claude-3-haiku@20240307") is prefixed with the Google publisher (publishers/google/models/). According to the information you've provided, the Claude 3 model is actually stored under the Anthropic publisher, so the correct model name should be prefixed with publishers/anthropic/models/.

Try changing the model name to "publishers/anthropic/models/claude-3-haiku@20240307" and see if that resolves the issue:

model = VertexAI(model_name="publishers/anthropic/models/claude-3-haiku@20240307")

As for accessing Anthropic models using the Model Garden, it's currently unclear whether LangChain supports this directly. The LangChain repository does not seem to contain any references to a langchain_anthropic module or similar, so it's possible that this functionality is not yet implemented or is part of a different package.

If changing the model name does not resolve your issue, or if you have further questions about accessing Anthropic models with LangChain, please provide more details about your use case and any error messages you're encountering. This will help me provide more accurate and helpful advice.

Sources

• Error accessing Claude 3 via Vertex AI
• libs/partners/anthropic/langchain_anthropic/init.py
• libs/community/tests/integration_tests/llms/test_vertexai.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nloeff]

I've made it work by changing the _BaseVertexAIModelGarden class:

    def endpoint_path(self) -> str:
        return self.client.endpoint_path(
            project=self.project, location=self.location, endpoint=self.endpoint_id
        ).replace("endpoints/","") # wrong default endpoint for claude 3 models

and calling:

llm = VertexAIModelGarden(project=PROJECT_ID, location=LOCATION, endpoint_id="publishers/anthropic/models/claude-3-haiku@20240307")

Does that work for you?

[vishal-jain-wwt]

I tried making this change to the endpoint_path function, which led to a new error -
InvalidArgument: 400 {"type":"error","error":{"type":"invalid_request_error","message":"messages: Field required"}}

[zakaria-aabbou]

This worked for me:

from langchain_google_vertexai.model_garden import ChatAnthropicVertex

# Initialize the ChatAnthropicVertex object
llm = ChatAnthropicVertex(
    model_name="claude-3-opus@20240229",
    project=PROJECT_ID,
    location=LOCATION
)

# Call the invoke method with your input text
response = llm.invoke("Hello, how are you?")

# Print the response
print(response.content)

Hello! As an AI language model, I don't have feelings, but I'm functioning properly and ready to assist you. How can I help you today?

[romaintoub]

for gemini or PaLM 2 models, you can import VertexAI eg. from langchain_google_vertexai import VertexAI