Python version pin

[szkafander]

Hi! Are there any near-term plans to relax the Python 3.8.16 version pin? As the 3.8 Security Supports ends in a year, users might need to upgrade. Many existing packages have stopped supporting 3.8 already.

Based on my testing, Covalent cannot handle minor version mismatch between the executor and lattice environments as it leads to incorrect unpickling. Are there any immediate workarounds?

[kessler-frost]

Hi @szkafander ! Thanks for the discussion. I was wondering if there's any specific executor that you found the issue with. I have been using Covalent on 3.10 for some time now and it seems to be working fine so far, that said I haven't been using the remote executors like Slurm that much so your input will be greatly appreciated.

[kessler-frost]

Also, is there a specific place that you found the 3.8.16 pin? I might be missing something, but I don't think we're pinning any python version for Covalent's requirements.

[szkafander]

Thanks for the reply! I am using the AWS Lambda and Batch executors. The base executor images are all Python 3.8.16 - any other minor version on the frontend caused data corruption in my testing. Specifically, when there is a Python version mismatch between the frontend and remote executors, when passing e.g., lists as arguments to the Electrons, the values are just read incorrectly by the remote machine.

If there is an easy way to upgrade the executor images to 3.10 or fix this issue another way, I’d be very interested to know!

[AlejandroEsquivel]

@szkafander I see you are trying to use the cloud executors with more version of python this is a known limitation however there are workarounds today you can leverage to get around this.

The high-level answer is that you can build your own base images with any python version.

AWS Batch Executor:
For AWS Batch, it currently uses a public ecr image we built and provide for all users. It is based on the Dockerfile living in the https://github.com/AgnostiqHQ/covalent-aws-plugins repo.
Building it would just be supplying new build args:

DOCKER_BUILDKIT=1 docker build --platform linux/amd64 --build-arg COVALENT_BASE_IMAGE=python:3.10-slim-bullseye --build-arg COVALENT_PACKAGE_VERSION="covalent>=0.202.0,<1" --tag covalent-executor-base .

Note that you can also set COVALENT_PACKAGE_VERSION to a specific covalent version as COVALENT_PACKAGE_VERSION="covalent==0.220.0" which is typically best if it is the same covalent version that you have locally.

Once you build this image you will need to upload it to either ECR and Dockerhub.

As our latest stable release of batch doesn't yet support custom base executor images you would likely need to either:

1. Edit the package in site-packages and update the hardcoded COVALENT_EXEC_BASE_URI variable to point to the newly uploaded image.
2. Re-install the plugin from source, cloning the aws-batch plugin repo and installing with pip install -e . and then setting the environment variable COVALENT_EXEC_BASE_URI

From there you should be able to dispatch a workflow using the custom base executor image. Also this is a great way to optimize the running of your workflow as you can bake in additional pip or other dependencies here so you can avoid using DepsPip.

AWS Lambda Executor:
For the lambda executor things are a bit different as lambda cannot pull from public image registries, essentially the image has to be a private AWS ECR repository in the same account as the lambda (for simplicity) our docs actually provide a very detailed outline how to build (very similar to batch), deploy the image to ECR and the lambda.

Recommend reading: https://docs.covalent.xyz/docs/user-documentation/api-reference/executors/awslambda

For lambda you can use the current stable version of the plugin unlike batch.

I hope this helps, let me know if you have any further questions and if you want me to provide further terraform scripts that may help as we will soon be making these public. Thanks for your interest in using covalent! Cheers :)

[santoshkumarradha]

Also note that proving custom containers is something the team is actively working on supporting from next (or worst case the next to that) versions of covalent.