AWS Lambda stream read timeout

[dazza-codes]

The code to call lambda and read a response has been working fine for most workloads. I ran into this while load testing with some modest lambda payloads and longer runtimes; approx 350 Mb and 25 seconds runtime (not too crazy).

The code is at https://github.com/dazza-codes/aio-aws/blob/master/aio_aws/aio_aws_lambda.py#L145-L221 and that module has a simple main example. The version details are below and aiobotocore is at 1.2.1

I've tried to remember and retrace the deeply nested creation of a client to figure out how to configure and/or pass some kind of timeout object all the way down to the aiohttp session, but I can't find it. When using aiohttp directly it's something like the following link notes [1], but what would be the same kind of thing for aiobotocore and could this be documented (or did I miss it already)?

[1] - https://docs.aiohttp.org/en/stable/client_quickstart.html#timeouts

Is there any way to pass any timeout args to a steam.read() call for a lambda response?

            response_payload = response.get("Payload")
            if response_payload:
                async with response_payload as stream:
                    data = await stream.read()

traceback:

/opt/conda/envs/gis/lib/python3.7/site-packages/aio_aws/aio_aws_lambda.py:170: in invoke
    await self.read_response()  # updates self.content
/opt/conda/envs/gis/lib/python3.7/site-packages/aio_aws/aio_aws_lambda.py:214: in read_response
    data = await stream.read()
/opt/conda/envs/gis/lib/python3.7/site-packages/aiohttp/streams.py:370: in read
    block = await self.readany()
/opt/conda/envs/gis/lib/python3.7/site-packages/aiohttp/streams.py:392: in readany
    await self._wait("readany")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <StreamReader e=ServerTimeoutError('Timeout on reading data from socket')>, func_name = 'readany'

    async def _wait(self, func_name: str) -> None:
        # StreamReader uses a future to link the protocol feed_data() method
        # to a read coroutine. Running two read coroutines at the same time
        # would have an unexpected behaviour. It would not possible to know
        # which coroutine would get the next data.
        if self._waiter is not None:
            raise RuntimeError(
                "%s() called while another coroutine is "
                "already waiting for incoming data" % func_name
            )
    
        waiter = self._waiter = self._loop.create_future()
        try:
            if self._timer:
                with self._timer:
>                   await waiter
E                   aiohttp.client_exceptions.ServerTimeoutError: Timeout on reading data from socket

/opt/conda/envs/gis/lib/python3.7/site-packages/aiohttp/streams.py:306: ServerTimeoutError

versions

$ pip show aiobotocore
Name: aiobotocore
Version: 1.2.1
Summary: Async client for aws services using botocore and aiohttp
Home-page: https://github.com/aio-libs/aiobotocore
Author: Nikolay Novik
Author-email: [email protected]
License: Apache 2
Location: /opt/conda/envs/gis/lib/python3.7/site-packages
Requires: aioitertools, wrapt, botocore, aiohttp
Required-by: aio-aws

$ pip show aiohttp
Name: aiohttp
Version: 3.7.4
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: [email protected]
License: Apache 2
Location: /opt/conda/envs/gis/lib/python3.7/site-packages
Requires: chardet, typing-extensions, multidict, async-timeout, attrs, yarl
Required-by: aiobotocore

$ python --version
Python 3.7.10

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

[thehesiod]

can use the timeouts from botocore:

        aioboto_session = aiobotocore.session.AioSession()
        config = botocore.client.Config(connect_timeout=10, read_timeout=10)
        async with aioboto_session.create_client('s3', config=config) as s3_client, \

[dazza-codes]

It could really help if the pydocs for aioboto_session.create_client provided more tips about the useful args and kwargs.

[dazza-codes]

See also https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

It would help if IDE tool tips and/or code completion were working better for these objects. It seems like the nested factory creation patterns obscure things that are otherwise common place.

A snippet to set or update the session default client config:

default_config = botocore.client.Config(
    connect_timeout=20,
    read_timeout=120,
    max_pool_connections=25,
    region_name="us-west-2"
)

# aiosession.get_default_client_config() is Optional[botocore.client.Config]
client_config = aiosession.get_default_client_config()
if client_config:
    client_config = client_config.merge(default_config)
else:
    client_config = default_config

aiosession.set_default_client_config(client_config)

[thehesiod]

pretty much all the aiobotocore methods follow the botocore docs, so you can use the botocore docs, no point in us replicating all their docs.

[thehesiod]

o i c you're saying so like help(method) would work, ya we should have an issue for that, but that's going to be a massive change I think.

[dazza-codes]

Related to this - updated some client creation functions in dazza-codes/aio-aws#13 - in that PR, there is examples of some pydocs with sphinx code examples to illustrate how to pass some optional args or kwargs and a link to the botocore docs. Most that project code uses type hints that help too. It seems like PyCharm does not pick up on the rtype annotations in aiobotocore very well (or maybe that's just my issue with some PyCharm indexing or settings or something.) It provides useful tool-tip docs in PyCharm, e.g.

[thehesiod]

ideally we find a way to forward the help docs across á la wrapt

[thehesiod]

btw i tested via help(aioboto method/class) and it seems to work correctly. It will go to the parent class for docs