Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor s3fs (and the aotbotocore) out as a dependency #43

Open
stijnvanhoey opened this issue May 22, 2023 · 0 comments
Open

Refactor s3fs (and the aotbotocore) out as a dependency #43

stijnvanhoey opened this issue May 22, 2023 · 0 comments

Comments

@stijnvanhoey
Copy link
Collaborator

stijnvanhoey commented May 22, 2023

s3fs is a convenient package to interact with s3 (feels like coding against normal file system), but since s3f3 relies on the async aiobotocore package some issues arised:

  • due to very strict dependencies in both boto3 and aiobotocore, the pip solver was not able to solve this unless defining boto3 as an additional install from s3fs (see daeb285 for fix)
  • aiobotocore requires raw_headers which are not according to specs in the moto-library used in the testing (see aiobotocore AttributeError - missing raw_headers getmoto/moto#3259 for description and 802b9f5 for the fix)
  • aiobotocore/s3fs keep in an recursive loop copy-pasting the files in the pytest-context leasing to an ever increasing file size when running the tests. Has been solved by relying on boto3 instead for the download, see 28a5e42

Hence, these issues have been handled in https://github.com/enram/vptstools/tree/SVH-country-filter, but I'm not sure if the convenience of s3fs is worthwhile having it as a dependency. It might be that excluding s3fs would make things easier to maintain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant