ModuleNotFoundError in AWS Python 3.12 images

[bassicallychris]

When using aws-psycopg2 in the AWS public.ecr.aws/lambda/python:3.12 image the error ModuleNotFoundError: No module named 'psycopg2._psycopg' is produced when using the sqlalchemy package.

[alexandreczg]

@bassicallychris Without offering a solution using this library, when using public.ecr.aws/lambda/python:3.12 I just use the psycopg2-binary

[bassicallychris]

Thanks for the response @alexandreczg. According to the psycopg2 documentation psycopg2-binary isn't meant for production use.

If you are the maintainer of a published package depending on psycopg2 you shouldn’t use psycopg2-binary as a module dependency. For production use you are advised to use the source distribution.

ref psycopg2 documentation

Ideally we are able to leverage a version of psycopg2 for aws-lambdas that is based on the source distribution.

[alexandreczg]

ah, that's good to know, haven't read that advisory. I guess gotta find a way to build it myself, seems like this lib here has lost support.

[ewenmcneill]

For anyone else finding this, the binaries (last updated July 2022) were built with GitHub Actions (see generally https://github.com/AbhimanyuHK/aws-psycopg2/tree/master/.github/workflows), and then uploaded (but after an accidental upload early in the testing of the actions I think the uploads are not automatic on every commit).

It looks like at the time of last build Python 3.6 to Python 3.10 were available for auto-building, so those are the ones included:

ewen@basadi:~/Downloads$ unzip -v aws_psycopg2-1.3.8-py3-none-any.whl | awk '/psycopg.cpython.*x86_64-linux/ {print $8;}' | sort -n -t- -k2
psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so
psycopg2/_psycopg.cpython-37m-x86_64-linux-gnu.so
psycopg2/_psycopg.cpython-38-x86_64-linux-gnu.so
psycopg2/_psycopg.cpython-39-x86_64-linux-gnu.so
psycopg2/_psycopg.cpython-310-x86_64-linux-gnu.so
ewen@basadi:~/Downloads$ 

Python 3.9 is supported until 2025, and Python 3.10 is probably supported until 2026. And the last batch of binary updates came when the built versions were close to expiry, so it might be a while before this package is updated to later Python versions.

That said, if you need Python 3.11 / Python 3.12 features, you can probably follow the steps the GitHub Workflow uses to build a later binary wheel locally.

Personally I'm going to stick with Python 3.10 for now, as I don't have anything that needs features only in Python 3.11/3.12.

Ewen