-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request]: Enhance Error Handling in FileSystem Imports to Improve Troubleshooting #31218
Comments
I like your idea. Thanks for opening this issue. |
filesystem extensions [gcp],[s3],etc are optional dependencies of beam. #31219 will cause excessive warning raised if user not intended to install these dependencies. Can we improve the error message you referred in the description instead?
to, e.g.
|
Even better, currently it hints |
@Abacn Thank you for reviewing my proposal. 👍
In my PR, there should not be any warning logs for modules ([gcp], [aws], etc.) that the user has not intended to installed. The import statement for not-installed modules should throw
However, on the other hand:
I think these ideas are excellent! I was considering this issue under the scope of "when modules (gcp/aws/azure) installed by the user as Filesystem fail to initialize due to for some reason (such as OpenSSL)." |
What would you like to happen?
I would like to make the import of Filesystem, which is defined in the top-level code of
apache_beam.io.filesystems
, easier to troubleshoot.https://github.com/apache/beam/blob/v2.56.0/sdks/python/apache_beam/io/filesystems.py#L36-L59
AS-IS:
PROPOSAL:
For context, I encountered a problem when launching a Beam job on CentOS 7 with apache-beam[gcp]==2.55.0 installed. The error occurs at the time of job initiation and is not an issue that occurs during job execution.
The error itself occurs on this line and is due to the failure to load
GCSFileSystem
at module initialization. This, in turn, is becauseGCSFileSystem
relies on therequests
package which, from version 2 onwards, requires OpenSSL 1.1.1 due to OS dependencies. CentOS 7 has OpenSSL 1.0.2 installed, so the behavior has changed with Beam version 2.55.0 and later. (This is not essential, so I have not investigated in detail.)I was able to resolve this quickly because I happened to know about these circumstances, but considering the future, it seems better to handle
ImportError
not just by suppressing it, but by logging a warning error.I can send a Pull Request. However, since it involves committing to a core area, I've raised an Issue first.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
The text was updated successfully, but these errors were encountered: