-
Notifications
You must be signed in to change notification settings - Fork 104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inference Component Name header is required #64
Comments
Jumpstart inference endpoints now need an InferenceComponentName |
@MithilShah |
Yes it did. Thanks! I modified kendra_chat_llama_2.py to this
|
@ecdedios I have the same issue. I am trying to understand what's the difference between endpoint_name and InferenceComponentName? If my sagemaker endpoint is meta-textgeneration-llama-2-7b-f-20240201-XXXXXX, what is point_name and what is InferenceComponentName? Appreciate your help with this! |
@utility-aagrawal I forgot exactly which one I'd used for inferenceComponentName but it's either the endpoint name or the model name. Here are some screenshots. Basically, you get the model name by clicking on the endpoint name. ![]() ![]() |
I am working on a fix , but @ecdedios is right. The one starting with "jumpstart.." is the endpoint name and the one in the "model" section is the Inference Component name. Testing the fix, but will release soon |
Thanks @ecdedios @MithilShah ! I tried with InferenceComponentName in endpoint_kwargs and got this error: ValueError: Error raised by inference endpoint: An error occurred (ValidationError) when calling the InvokeEndpoint operation: Inference Component Name header is not allowed for endpoints to which you dont plan to deploy inference components. Please remove the Inference Component Name header and try again. It just worked without InferenceComponentName. It's weird because the same code wasn't working yesterday and was asking me to include InferenceComponentName. I am not sure what's changed since yesterday. |
@utility-aagrawal can you please try again. I have added a new variable. If you deploy the endpoint via the console, it deploys the model to an InferenceComponent and you need to specify a INFERENCE_COMPONENT_NAME environment variable. However, if you deploy via the SDK you have to option of deploying directly via the endpoint without using an inferencecomponent. If you do that, just ignore the INFERENCE_COMPONENT_NAME environment variable. |
Thanks @MithilShah ! I'll try and let you know. |
I'm getting the following error
botocore.errorfactory.ValidationError: An error occurred (ValidationError) when calling the InvokeEndpoint operation: Inference Component Name header is required for endpoints to which you plan to deploy inference components. Please include Inference Component Name header or consider using SageMaker models.
when I run
python kendra_chat_llama_2.py
Name: boto3
Version: 1.34.21
The text was updated successfully, but these errors were encountered: