ModuleNotFoundError with AWS lambda

[BatArni]

Hi,

I am relatively new to lithops and aws lambda, so pls apologise if my question is elementary. I managed to get lithops "map" working with aws lambda for a simple function that does not have any dependencies. When I modify the function to do something with pandas, however, I can run the function without issues in "localhost" mode but I get the error "ModuleNotFoundError: No module named 'pandas'" when using the 'serverless' mode.
My understanding was that lithops would manage the dependencies when deploying the function to aws but maybe that's not correct? Could you please guide me a bit how the fix this issue?

Lithops / python version

Lithops v3.1.0 - Python3.11

Error message:

Traceback (most recent call last):
File "/var/task/lithops/worker/jobrunner.py", line 209, in run
File "/opt/python/cloudpickle/cloudpickle.py", line 457, in subimport
ModuleNotFoundError: No module named 'pandas'

Many thanks!

[JosepSampe]

Hi @BatArni,

The feature of automatically transferring libraries is specially designed for smaller libraries. In the case of larger libraries like pandas, which require tens of other libraries simultaneously, it is recommended to install the library in the runtime instead of depending on Lithops to transfer it. The reason for this is that, in the case of pandas, the package to be transferred can potentially exceed 50MB. Relying on Lithops to transfer it will significantly slow down function invocation and execution.

To create your own custom container runtime in AWS you can follow these instructions

[BatArni]

Hi @JosepSampe ,

Firstly, thanks for the quick response! Guess I will have to try to make it work with the custom container as suggested. In the meantime, is there by any chance anything I could do to rely on Lithops for the deployment even if it would slow down invocation/execution?

[rohitsharma2790]

Hi @BatArni , You can package a pandas library and deploy it as AWS layer in AWS lambda. That layer can be called in your code.

[BatArni]

Thanks @rohitsharma2790 for the suggestion, will try this out as well.