Convert staging job to python for forecast-only mode #2651
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Change the extension of the exglobal_stage ex-script from "sh" to "py". Refs NOAA-EMC#2475
- Update script extension for ex-script from "sh" to "py". - Pull COM* variable declares up from ex-script. Refs NOAA-EMC#2475
Remove the functions and calls to set up symlinks to ICs in ROTDIR. Refs NOAA-EMC#2475
- Add initial new yaml files for staging information - Add new stage.py to python tasks. - Add first draft pythonization of stage ex-script. Much more work is still to be done. Refs NOAA-EMC#2475
Revert changes to workflow/setup_expt.py; will do in later task Refs NOAA-EMC#2475 * upstream/develop: Sea-ice analysis insertion (NOAA-EMC#2584) Refactored archiving (NOAA-EMC#2491) Add remove RUNDIRS step in CI before creating experements (NOAA-EMC#2607)
Also fix to set target and remove source Refs NOAA-EMC#2475
Add target, remove source, and update file info Refs NOAA-EMC#2475
Update to use mkdir and copy instead of target and required Refs NOAA-EMC#2475
If RUN=gefs add keys_gefs to keys. Refs NOAA-EMC#2475
Set to .false. by default; needed for staging job Refs NOAA-EMC#2475
- remove master yaml, no longer using - update fv3_cold, ice, ocean, and wave yamls Refs NOAA-EMC#2475
- Add keys for GEFS - Cleanup Refs NOAA-EMC#2475
- General cleanup - Rework determine_stage function - Rework execute_stage function Refs NOAA-EMC#2475
- Delete da.yaml.j2; will remake in follow-up work - Update fv3_warm.yaml.j2 to not use src/head variables Refs NOAA-EMC#2475
KateFriedman-NOAA
requested review from
WalterKolczynski-NOAA and
DavidHuber-NOAA
Add extra whitespaces in exglobal_stage_ic.py to address E231 error. Refs NOAA-EMC#2475
ush/python/pygfs/task/stage.py
Outdated
|
|
||
| for set_yaml in stage_sets: | ||
|
|
||
| stage_set = parse_j2yaml(os.path.join(stage_parm, set_yaml), stage_dict) |
There was a problem hiding this comment.
If all variables need to be defined in the jinja file, then you might consider adding allow_missing=False to the parse_j2yaml call. This will raise an exception if a variable referenced in the template is not defined.
Improve readability of the stage yamls Refs NOAA-EMC#2475
|
Got some good feedback from reviewers (offline and online). Will ponder feedback and work on updates. Will leave this draft PR open while working on it. Thank you all so far! |
Also add ICSDIR to config.base Refs NOAA-EMC#2475
Also update wave COM variable Refs NOAA-EMC#2475
|
Will be making some iterative commits. Please do not rereview until rerequested. |
- set job for gdas_half cycledef only - update fcst jobs dependencies Refs NOAA-EMC#2475
- Delete the fill_ROTDIR functions used in half cycle setup; functions are replaced by half cycle stage_ic jobs - Update --icsdir setup flag to allow user provided path for all RUNs Refs NOAA-EMC#2475
* origin/develop: Add observation preparation job for aerosols DA to workflow (NOAA-EMC#2624) Remove ocean daily files (NOAA-EMC#2689) Update Jenkinsfile Add Hercules-EMC to the Jenkins configurable parameter list (NOAA-EMC#2685) Update gdas.cd and gsi_utils hashes (NOAA-EMC#2641) Add ability to use GEFS replay ICs (NOAA-EMC#2559) Replace `sleep` with `wait_for_file` (NOAA-EMC#2586) Add COM template for JEDI obs (NOAA-EMC#2678) Link both global-nest fix files and non-nest ones at the same time (NOAA-EMC#2632) Update ufs-weather-model (NOAA-EMC#2663) Add ability to process ocean/ice products specific to GEFS (NOAA-EMC#2561) Update cleanup job to use COMIN/COMOUT (NOAA-EMC#2649) Add overwrite to creat experiment in BASH CI (NOAA-EMC#2676) Add handling to select CRTM cloud optical table based on cloud scheme and update calcanal_gfs.py (NOAA-EMC#2645) Refs NOAA-EMC#2475
Makes yamls more generic and uses paths set previously Refs NOAA-EMC#2475
- Add new keys - Remove determine function - Update execute function to use single stage.yaml.j2 Refs NOAA-EMC#2475
- Add RDATE and DTG_PREFIX - Remove USE_OCN_PERTURB_FILES - Always declare COM_ATMOS_ANALYSIS Refs NOAA-EMC#2475
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR converts the staging job from shell to python. The existing forecast-only
stage_icis converted to use python and yaml via jinja parsing. This PR does not impact the existingROTDIRsymlink population done for cycled mode via the setup scripts.Changes in this PR:
scripts/exglobal_stage_ic.shtoscripts/exglobal_stage_ic.py.jobs/JGLOBAL_STAGE_ICto use.pyscript extension. MoveGDATE/gPDY/gcycsettings up to JJOB from ex-script, as well as move upCOM*variable declarations andMEMDIR[_ARRAY]settings.PYTHONPATHexport tojobs/rocoto/stage_ic.sh.parm/stagefolder to hold newly created yamls (*.yaml.j2) for each initial condition set currently handled in existingscripts/exglobal_stage_ic.sh. Included yamls for warm start (fv3_warm.yaml.j2) andDO_NEST=YES(fv3_nest.yaml.j2) to retain functionality in existingscripts/exglobal_stage_ic.shbut have not tested these configurations in forecast-only mode.ush/python/pygfs/task/stage.pyto house staging job python functions for call fromscripts/exglobal_stage_ic.py.export USE_OCN_PERTURB_FILES=".false."to gfsconfig.base; need variable as key in staging job python regardless ofRUN.stage_icjob rocoto dependencies from xml. Do not need and removes area of duplicate maintenance.Follow-up PRs will add
stage_icjob to cycled mode and extended capabilities as needed as part of issue #2475.Resolves #2650
Type of change
Change characteristics
How has this been tested?
Ran the forecast-only CI tests on Hera:
Outputs match CI tests run from
develop. Logs available for review.Snippet from C48 S2SWA GEFS
stage_icjob log showing themkdirand copy ofmem002files intoROTDIR: