Develop into Main (1.12.0) (#114)

* Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <[email protected]> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <[email protected]> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <[email protected]> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <[email protected]> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <[email protected]> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <[email protected]> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <[email protected]> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <[email protected]> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <[email protected]> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: sureshshsv <[email protected]> Co-authored-by: sureshshsv <[email protected]> * added README information and updates (#113) * fixed pymock issues... again Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: sureshshsv <[email protected]> Co-authored-by: sureshshsv <[email protected]>
podaac · Dec 12, 2022 · 99807eb · 99807eb
1 parent b9090a6
commit 99807eb
Show file tree

Hide file tree

Showing 9 changed files with 228 additions and 101 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,12 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
+## 1.12.0
+### Fixed
+- Added EDL based token downloading, removing CMR tokens [98](https://github.com/podaac/data-subscriber/issues/98),
+### Added
+- Added ability to download by filename [109](https://github.com/podaac/data-subscriber/issues/109) and additional regression testing
+
 ## 1.11.0
 ### Fixed
 - Fixed an issue where token-refresh was expecting a dictionary, not a list of tuples

diff --git a/Downloader.md b/Downloader.md
@@ -6,7 +6,7 @@ For installation and dependency information, please see the [top-level README](R
 
 ```
 $> podaac-data-downloader -h
-usage: PO.DAAC bulk-data downloader [-h] -c COLLECTION -d OUTPUTDIRECTORY [--cycle SEARCH_CYCLES] [-sd STARTDATE] [-ed ENDDATE] [-f] [-b BBOX] [-dc] [-dydoy] [-dymd] [-dy] [--offset OFFSET] [-e EXTENSIONS] [--process PROCESS_CMD] [--version] [--verbose] [-p PROVIDER] [--limit LIMIT]
+usage: PO.DAAC bulk-data downloader [-h] -c COLLECTION -d OUTPUTDIRECTORY [--cycle SEARCH_CYCLES] [-sd STARTDATE] [-ed ENDDATE] [-f] [-b BBOX] [-dc] [-dydoy] [-dymd] [-dy] [--offset OFFSET] [-e EXTENSIONS] [--granule-name GRANULE] [--process PROCESS_CMD] [--version] [--verbose] [-p PROVIDER] [--limit LIMIT]
 
 optional arguments:
  -h, --help show this help message and exit
@@ -34,6 +34,8 @@ optional arguments:
  --offset OFFSET Flag used to shift timestamp. Units are in hours, e.g. 10 or -10.
  -e EXTENSIONS, --extensions EXTENSIONS
  The extensions of products to download. Default is [.nc, .h5, .zip, .tar.gz]
+ -gr GRANULE, --granule-name GRANULE
+ The name of the granule to download. Only one granule name can be specified. Script will download all files matching similar granule name sans extension.
  --process PROCESS_CMD
  Processing command to run on each downloaded file (e.g., compression). Can be specified multiple times.
  --version Display script version information and exit.
@@ -121,6 +123,20 @@ machine urs.earthdata.nasa.gov
 
 ## Advanced Usage
 
+### Download data by filename
+
+If you're aware of a file you want to download, you can use the `-gr` option to download by a filename. The `-c` (COLLECTION) and `-d` (directory) options are still required.
+
+The `-gr` option works by taking the file name, removing the suffix and searching for a CMR entry called the granuleUR. Some examples of this include:
+
+| Collection | Filename | CMR GranuleUR |
+| --- | ----------- | ----------- |
+| MUR25-JPL-L4-GLOB-v04.2 | 20221206090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2.nc | 20221206090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2 |
+| JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F | S6A_P4_2__HR_STD__NR_077_039_20221212T181728_20221212T182728_F07.nc | S6A_P4_2__HR_STD__NR_077_039_20221212T181728_20221212T182728_F07 |
+
+Because of this behavior, granules without data suffixes and granules where the the UR does not directly follow this convention may not work as anticipated. We will be adding the ability to download by granuleUR in a future enhancement.
+
+
 ### Download data by cycle
 
 Some PO.DAAC datasets are better suited for cycles based search instead of start and end times. To enabled this, we've added 'cycle' based downloading to the data-downloader. The following example will download data from cycle 42:

diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "podaac-data-subscriber"
-version = "1.11.0"
+version = "1.12.0"
 description = "PO.DAAC Data Subscriber Command Line Tool"
 authors = ["PO.DAAC <[email protected]>"]
 readme = "README.md"