You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a GCSDownloader that can fetch the data from Google Cloud Storage. It should support an authentication token, ideally with the option to read it from an environment variable.
See matched feature requests for other cloud storage services from Amazon's AWS (#363) and Microsoft's Azure (#382).
This would require:
A new downloader (GCSDownloader) in pooch/downloaders.py (see https://www.fatiando.org/pooch/latest/downloaders.html and the existing downloaders). Make sure to add it to the choose_downloader function so that Pooch can automatically find it based on the prefix (gs).
The test data in our data folder uploaded to the storage so we can test that it works.
Tests in pooch/tests/test_downloaders.py that check if the download works and that any errors that should be raised are actually raised.
I've got a fully functional GCSDownloader class here in a fork, but minus the testing. It uses the google-cloud-storage package for authentication/downloading, which can be passed as a token to the downloader or read from an environment variable. It allows usage of the tqdm progress bar option.
# Authorize by setting an environment variableimportosimportpoochcredentials="google_app_credentials.json"url="gs://bucket_name/blob_name.txt"os.environ["GOOGLE_APPLICATION_CREDENTIALS"] =credentialsfilename=pooch.retrieve(url, known_hash=None)
# Authorize by passing credentials to custom downloaderfrompoochimportGCSDownloadercredentials="google_app_credentials.json"downloader=GCSDownloader(credentials=credentials)
filename=pooch.retrieve(url, known_hash=None, downloader=downloader)
I can't speak to long-term maintenance, but I would be interested in adding tests and submitting a PR within the next month.
The text was updated successfully, but these errors were encountered:
Add a GCSDownloader that can fetch the data from Google Cloud Storage. It should support an authentication token, ideally with the option to read it from an environment variable.
See matched feature requests for other cloud storage services from Amazon's AWS (#363) and Microsoft's Azure (#382).
This would require:
I've got a fully functional GCSDownloader class here in a fork, but minus the testing. It uses the
google-cloud-storage
package for authentication/downloading, which can be passed as a token to the downloader or read from an environment variable. It allows usage of the tqdm progress bar option.I can't speak to long-term maintenance, but I would be interested in adding tests and submitting a PR within the next month.
The text was updated successfully, but these errors were encountered: