Exposing more boto3 client options to ChatBedrock and doc example?

[P1llus]

With the addition of the opus model support on Bedrock, which generally tends to always hit the default timeout options, would it be okay to add the different retry/timeout options from boto3 as common options to ChatBedrock?

Currently we have support for a few common keys here: https://github.com/langchain-ai/langchain-aws/blob/main/libs/aws/langchain_aws/llms/bedrock.py#L300.

However just changing from Claude 2/2.1 or Claude 3 Sonnet to Opus increases the runtime mostly by 10x (mine are consistently around 20-35 seconds on old models to 200+ seconds with Opus)..
UPDATE:
Later it also turns out many of the timeouts reported by boto3 was actually general throttling (as seen in bedrock logs), however there was no logs or errors indicating any throttling happening, just reported as a timeout.
Changing the timeout settings + upping the max_attempts was necessary to even get it to retry properly it seems.

I would assume that once more people start using this new library, and depending on the popularity of Opus, a few changes are recommended:

1. Adding a higher default timeout, to prevent common issues to appear.
2. Adding more boto3 client options available as arguments to ChatBedrock/BedrockBase, so that developers can instantly see them as a supported list of arguments when developing using this chat model.
3. A small documentation example that links to the available boto3 options that would work with Bedrock, and a small example on how to create a custom Botocore.config.Config() object that can be passed to the existing config argument would be greatly appreciated. I did get it working through trial and error, but only from boto3 examples directly and not from langchain issues or docs.

[3coins]

@P1llus
All great ideas, there is plan to abstract out the client creation to a separate module with more options. Note that users can create their own client and pass in to the Bedrock classes, which supports all options available with boto3. Defaulting to a higher timeout might make sense for Opus, but not applicable universally. If there is a config that you have worked out which can work universally, open a PR with that change, and we can review it.
Agree that we need more examples and documentation, we welcome contributions from the community, and let us know if you need any help with this.

[P1llus]

@3coins Ah yeah that is true about the defaults, might be a bit overkill to change it like that.

I think something in-between would be if its possible for us to detect the underlying issue with some sort of logging.
As mentioned above, the only message received from Chat bedrock is that there was a timeout.
However it was actually hitting a throttle and 504's with a backoff.
There was no indication on that from the client perspective, and only the logs from the bedrock service itself would highlight it.

I am unsure if this is how boto3 logs it as well or if its the ChatBedrock class that don't surface information like that?

[3coins]

Yes, totally makes sense, exception handling is not the best at the moment and we can improve that. I have to check if there is a separate exception thrown by the API that we can surface here.