Use in AWS Lambda with minimal layer sizes #833
Labels
Comments
|
The docker images for lambci contain versions of boto3 that are not currently consistent with the published documentation. published: lambci container: The To check it ... Using this lambda handler: # lambda_versions.py
"""
Lambda Library Versions
***********************
This lambda function handler prints built-in package versions.
"""
import boto3
import botocore
import docutils
import jmespath
import dateutil
import s3transfer
import six
import urllib3
import os
import pprint
import sysconfig
def lambda_handler(event, context):
os.system("ls -1d /var/runtime/*.dist-info | sort")
print()
sys_paths = sysconfig.get_paths()
pprint.pprint(sys_paths)
print()
versions = {
"boto3": boto3.__version__,
"botocore": botocore.__version__,
"docutils": docutils.__version__,
"jmespath": jmespath.__version__,
"dateutil": dateutil.__version__,
"s3transfer": s3transfer.__version__,
"six": six.__version__,
"urllib3": urllib3.__version__,
}
pprint.pprint(versions)
print()
# pip requirements format
for k, v in versions.items():
print(f"{k}=={v}")
print()
return {"statusCode": 200, "body": "lambda versions done"}WIth a current python3.6 lambci container: It reports the latest container built-in library versions (although these may not be what is actually running on AWS Lambda at the time of writing): |
This was referenced Nov 3, 2020
|
btw lambci is used under the hood in moto, I actually did a lot of work ensuring that would work |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It would help if the test-CI matrix and the setup dependencies supported the current versions of AWS Lambda for boto3 and botocore. It would be useful if one cell in a test-CI matrix pinned the boto3/botocore versions to the exact patch versions used in the current AWS Lambda runtime versions (and possibly ran in a container using the lambci runtime used by AWS Lambda for each python version in the runtimes available; excluding anything that does not support asyncio, of course). At the time of writing, the setup.py does not allow a range of botocore or boto3 versions that are compatible with the current AWS Lambda versions; the current AWS Lambda runtime versions are listed in [1] as follows.
[1] https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
The
setup.pycontains the following at the time of writing:It could help if
setup.pycould keep track of AWS Lambda runtime versions and allow a range of versions that are compatible, e.g. maybe (?)Of course, doing so would entail that a test-CI matrix uses the actual AWS Lambda pinned versions to check that everything works OK with those releases. Then it should be possible to easily and confidently use aiobotocore in AWS Lambda layers and functions.
It might be interesting to try lambCI
Motivation for Specific AWS Lambda Compatibility
Of course, it is always possible to bundle a different boto3/botocore into an AWS Lambda layer or function and carry on regardless of whatever the AWS Lambda runtime provides out of the box. However, sometimes a project may have so many dependencies or multiple layers that it exceeds the AWS package size limits [2]. In some serverless applications, the size of a lambda layer can approach the limits of layer sizes. In these cases, it helps to delete boto3 and all of its dependencies from a layer zip archive, because the lambda runtime container already provides them out-of-the-box. Doing so can save a big chunk of space in an AWS Lambda layer. The following example is the dep-tree for the current AWS Lambda boto3 version:
[2] https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html
The text was updated successfully, but these errors were encountered: