All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog
- Fixed bug where --subset in combination with the subscriber caused errors
- Added support for Harmony subsetting
- Added support for wildcard search patterns in podaac-data-downloader when executed with the -gr option (i.e. search/download by CMR Granule Ur/Id). Also, added usage details to Downloader.md to describe this new feature 138.
- Fixed an issue where a required library wasn't being included in the installation.
- Added --dry-run option to subscriber and downloader to view the files that would be downloaded without actuall downloading them. 102
- Added new feature allowing regex to be used in
--extension-eoptions. For example using -ePTM_\\d+would match data files likefilename.PTM_1,filename.PTM_2andfilename.PTM_10, instead of specifying all possible combinations (-e PTM_1, -e PTM_2, ..., -e PMT_10) 115 - Added check for updated version 70
- Removed CMR Token from log messages 127
- Added EDL based token downloading, removing CMR tokens 98,
- Added ability to download by filename 109 and additional regression testing
- Fixed an issue where token-refresh was expecting a dictionary, not a list of tuples
- Fixed issues where token was not propagated to downloader CMR query 94
- Fixed an issue with 503 errors on data download not being re-tried. 97
- added ".tiff" to default extensions to address #100
- removed erroneous 'warning' message on not downloading all data to close 99
- updated help documentation for start/end times to close 79
- Added citation file creation when data are downloaded 91. Required some updates to the regression testing.
- Fixed an issue where using a default global bounding box prevented download of data that didn't use the horizontal spatial domain 87
- Fixed limit option not being respected. 86
- Support for SHA-256 and SHA-512 checksums
- Changed minimum supported python version to 3.7, down from 3.8.
- Switched to poetry as the build tool for the project
- check if file exists before downloading a file. 17
- added automated regression testing
- Implemented Search After CMR interface to allow granule listings > 2000 15
- Retry CMR queries on server error using random exponential backoff max 60 seconds and 10 retries
- Refresh token if CMR returns 401 error
- Converted print statements to log statements
- limit to set limit of downloads- useful for testing
- cycle based downloads to the podaac-data-downloader. 41
- conftest.py added to force module inclusion for pytest
- podaac-data-downloader script for bulk data downloading
- created library of common access mechanisms to split between subscriber and downloader capabilities
- added .tar.gz to list of default extensions. 40
- Ignore error if destination directory already exists. 46
- Updated the naming convention of .update file. 44
- one of -m, -sd, or -ed must be given to subscriber. Previously -m 60 was the default if nothing was specified.
- use of ".update" file naming convention. This will still work, but will be renamed to .update__COLLECTIONNAME after a successful run. the ".update" file will need to be manually cleaned up. See issue 44
- issue where only specifying an end data cause issues in subscriber. 39
- Made number of files to download a non-verbose default printout. 33
- Auto build and deploy to pypi on tag/release.
- Added ability to call a process on downlaoded files. Thank to Joe Sapp.
- Turned -e option into 'additive' mode (multiple -e options allowed.) Thanks to Joe Sapp
- issue not being able to find granuleUR #28
- added warning for more than 2k granules
- strip newline characters from .update to fix #25
- added --offset flag for timestamp shift when creating DOY folder - (resolves #23)
- added ability to change the provider using the -p/--provider flag. Default is 'POCLOUD'
- added pyproject info and setup.py fixes to enable pypi pushes
- added pytest and flake8 fixes for automated builds
- changed changing created_at to updated_since to allow for re-download of updated granules based on collection redeliveries - (resolves #18)
- added additional non-flat output directory option of -dy - (resolves #13)
- Added logging capability using the SUBSCRIBER_LOGLEVEL environment variable
- Added -st and -ed flags and respect the .update flag
- removed the -ds flag as it caused confusion.
- added default layouts for non-flat output directories - (resolves #6)
- Added logging capability using the SUBSCRIBER_LOGLEVEL environment variable
- added additional non-flat output directory option of -dy - (resolves #13)
- updated urllib3>=1.26.5 (fixes https://github.com/advisories/GHSA-q2q7-5pp4-w6pg)
- User Agent to request so we can determine better support posture based on metrics
- data subscriber functionality