| # # # Distribution Statement A. Approved for public release. Distribution unlimited. | # # # | # # # Author: | # # # Naval Research Laboratory, Marine Meteorology Division | # # # | # # # This program is free software: you can redistribute it and/or modify it under | # # # the terms of the NRLMMD License included with this program. This program is | # # # distributed WITHOUT ANY WARRANTY; without even the implied warranty of | # # # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the included license | # # # for more details. If you did not receive the license, for more information see: | # # # https://github.com/U-S-NRL-Marine-Meteorology-Division/ Version 1.5.3 (2022-11-07) ************************** v1.5.3: 2022-11-07, update python install process, alternate command line args funcs, unique storm dirnames for invests, bug fixes 2022-11-04 bug fixes ==================== GEOIPS/geoips#108: 2022-11-04, bug fixes Documentation Updates --------------------- * Clarify steps in updating CHANGELOG during internal release process :: modified: CHANGELOG_TEMPLATE.md Test Repo Update ---------------- * Update abi.tc.Infrared.imagery_annotated metadata YAML * Add Add storm_start_datetime to metadata yaml :: modified: tests/outputs/abi.tc.Infrared.imagery_annotated/20200918_195020_AL202020_abi_goes-16_Infrared_110kts_100p00_1p0.png.yaml fix sun.set_time bug in overpass_predictor.py ============================================= GEOIPS/geoips#114: 2022-10-27, fix sun.set_time bug in overpass_predictor.py Bug fixes --------- Fix sun.set_time bug in overpass_predictor.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Previously only check sun.rise_time * sun.rise_time can be valid while sun.set_time is None, so must check both. * return None if either sun.rise_time or sun.set_time is None:: modified: geoips/sector_utils/overpass_predictor.py VIIRS lat/lon variables may contain unmasked fill values ======================================================== GEOIPS/geoips#113: 2022-10-27, VIIRS lat/lon variables may contain unmasked fill values Bug fix ------- modified: geoips/interface_modules/readers/viirs_netcdf.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Loop through xarray_return data types, and mask latitude/longitude by their fill value cowvr edge case - no usable sectored data ========================================= GEOIPS/geoips#111: 2022-10-27, cowvr edge case - no usable sectored data Bug fix ------- modified: geoips/interface_modules/procflows/config_based.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Add check if sectored data contains any usable data * Only add variable to all_vars list if sectored data set is not entirely comprised of NaNs modified: geoips/interface_modules/procflows/single_source.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Add check if sectored data contains any usable data * Only add variable to all_vars list if sectored data set is not entirely comprised of NaNs add sectored and unsectored AMSR2 winds outputs =============================================== GEOIPS/geoips_system_nrl#61: 2022-10-27, add sectored and unsectored AMSR2 winds outputs Major New Functionality ----------------------- Add sectored and unsectored products to product_inputs/amsr2.yaml ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Allows producing sectored and unsectored text winds outputs:: modified: geoips/yaml_configs/product_inputs/amsr2.yaml get_product allow empty dict, bug fixes ======================================= GEOIPS/data_fusion#8: 2022-10-21, get_product allow empty dict, bug fixes modified: geoips/dev/product.py ------------------------------- * Previously would only replace individual fields within dictionaries in product_inputs * Now, if an empty dictionary is included within product_inputs, it will replace the entire product_params dictionary with an empty dictionary. * This is useful when using product_templates, and replacing the algorithm or colormap, etc Test Repo Updates ----------------- Update ASCAT UHR test data path ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Renamed ASCAT UHR test data subdirectories with additional test cases:: modified: tests/scripts/ascat_uhr.tc.wind-ambiguities.imagery_windbarbs.sh Turn off minor ticks ^^^^^^^^^^^^^^^^^^^^ * matplotlib 3.6.0 sometimes has inconsistent results with including minor ticks or not. * Unclear why it impacts some colorbars and not others. * We may eventually add support for including minor ticks within mpl_colors_info, but for now * explicitly turn off minor ticks so outputs will continue to match (use the old default):: modified: geoips/image_utils/mpl_utils.py create unique storm dirnames for invests ======================================== GEOIPS/geoips#103: 2022-10-17, create unique storm dirnames for invests Major New Functionality ----------------------- geoips/interface_modules/filename_formats/tc_clean_fname.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Allow passing "output_dict" to allow using unique directory name for INVESTS geoips/interface_modules/filename_formats/tc_fname.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * def tc_fname * Allow passing output_dict to provide current output parameters for overall filename specifications * def assemble_tc_fname * Allow passing both "output_dict" and "sector_info" to allow timestamp in dirname for INVESTS geoips/interface_modules/filename_formats/utils/tc_file_naming.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Allow passing both "output_dict" and "sector_info" to support .%Y%m%d%H dirname for INVESTS * output_dict['file_path_modifications']['unique_invest_dirs'] True * storm_start_datetime is datetime object * sector_info['original_storm_start_datetime'] if it exists, else * sector_info['storm_start_datetime'] * storm number > 69 (ie, invest) * output_dict['file_path_modifications']['existing_invest_dirs_allowable_time_diff'] > 0 * If specified, use existing directory closest in time to storm_start_datetime * If none exist, use storm_start_datetime appended to INVEST directory * Ie, SH932020.2020020506 vs SH932020 * If SH932020.2020020406 exists, would use that rather than creating 2020020506 * SH162020 does NOT contain the extra storm start datetime information geoips/interface_modules/trackfile_parsers/bdeck_parser.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Add storm_start_datetime field to bdeck sector info * pull from first entry in bdeck file * Add original_storm_start_datetime field to bdeck sector_info * Pull from filename if available (since bdeck entries can change) * DO NOT INCLUDE in dictionary if it is not available * If it exists, this will be a more consistent value than storm_start_datetime (which can change with subsequent deck files) geoips/interface_modules/filename_formats/metadata_default_fname.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * def metadata_default_fname * Allow passing output_dict to provide current output parameters for overall filename specifications * def assemble_metadata_default_fname * Allow passing both "output_dict" and "sector_info" to allow timestamp in dirname for INVESTS geoips/interface_modules/filename_formats/text_winds_tc_fname.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * def text_winds_tc_fname * Allow passing output_dict to provide current output parameters for overall filename specifications * def assemble_text_winds_tc_fname * Allow passing both "output_dict" and "sector_info" to allow timestamp in dirname for INVESTS Test Repo Updates ----------------- Add storm_start_datetime to YAML metadata outputs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * modified: tests/outputs/abi.tc.IR-BD.imagery_annotated/20200918_195020_AL202020_abi_goes-16_IR-BD_110kts_100p00_1p0.png.yaml * modified: tests/outputs/abi.tc.Visible.imagery_annotated/20200918_195020_AL202020_abi_goes-16_Visible_110kts_100p00_1p0.png.yaml * modified: tests/outputs/amsr2.tc.89H-Physical.imagery_annotated/20200518_073601_IO012020_amsr2_gcom-w1_89H-Physical_140kts_100p00_res1p0-cr300.png.yaml * modified: tests/outputs/amsub_mirs.tc.183-3H.imagery_annotated/20210419_235400_WP022021_amsu-b_metop-a_183-3H_115kts_100p00_1p0.png.yaml * modified: tests/outputs/ascat_knmi.tc.windbarbs.imagery_windbarbs_clean/20210421_014248_WP022021_ascat_metop-c_windbarbs_120kts_78p20_0p5-clean.png.yaml * modified: tests/outputs/ascat_low_knmi.tc.windbarbs.imagery_windbarbs/20210421_014156_WP022021_ascat_metop-c_windbarbs_120kts_35p17_1p0.png.yaml * modified: tests/outputs/ascat_uhr.tc.wind-ambiguities.imagery_windbarbs/20210421_014200_WP022021_ascatuhr_metop-c_wind-ambiguities_120kts_100p00_0p1.png.yaml * modified: tests/outputs/gmi.tc.89pct.imagery_clean/20200917_172045_AL202020_gmi_GPM_89pct_115kts_78p16_res1p0-cr300-clean.png.yaml * modified: tests/outputs/hy2.tc.windspeed.imagery_annotated/20211202_084039_WP272021_hscat_hy-2b_windspeed_95kts_97p06_1p0.png.yaml * modified: tests/outputs/mimic_fine.tc.TPW-PWAT.imagery_annotated/20210419_230000_WP022021_mimic_tpw_TPW-PWAT_115kts_100p00_1p0.png.yaml * modified: tests/outputs/oscat_knmi.tc.windbarbs.imagery_windbarbs/20210209_025351_SH192021_oscat_scatsat-1_windbarbs_135kts_75p10_1p0.png.yaml * modified: tests/outputs/saphir.tc.183-3HNearest.imagery_annotated/20210209_003103_SH192021_saphir_meghatropiques_183-3HNearest_135kts_88p76_1p0.png.yaml * modified: tests/outputs/sar.tc.nrcs.imagery_annotated/20181025_203206_WP312018_sar-spd_sentinel-1_nrcs_130kts_58p51_res1p0-cr300.png.yaml * modified: tests/outputs/ssmi.tc.37pct.imagery_clean/20200519_080900_IO012020_ssmi_F15_37pct_110kts_50p65_1p0-clean.png.yaml * modified: tests/outputs/viirsday.tc.Night-Vis-IR.imagery_annotated/20210209_074210_SH192021_viirs_noaa-20_Night-Vis-IR_130kts_100p00_1p0.png.yaml Bug fixes --------- Do not attempt to set_ticks if cbar_ticks is not defined geoips/image_utils/mpl_utils.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Replace fig.savefig frameon=False argument with facecolor="none" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * frameon deprecated maplotlib v3.1.0, support removed v3.6.0 * facecolor="none" also works with 3.5.x * https://matplotlib.org/stable/api/prev_api_changes/api_changes_3.1.0.html?highlight=frameon * Updated files:: geoips/image_utils/mpl_utils.py geoips/interface_modules/output_formats/unprojected_image.py allow alternate command line args funcs ======================================= GEOIPS/data_fusion#8: 2022-09-29, allow alternate command line args funcs Enhancements ------------ new: tests/sectors/tc_bdecks/bwp142022.dat ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ modified: geoips/commandline/args.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Allow passing alternate check_args_func and get_args_func to get_command_line_args * Default output_format None vs imagery_annotated modified: geoips/commandline/run_procflow.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Allow passing alternate "get_command_line_args" func to run_procflow main modified: geoips/dev/product.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Add 'xarray_dict_to_output_format' product type * Allow specifying "product_template" within product_params YAML as well as product_inputs modified: geoips/image_utils/mpl_utils.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Support additional mpl_colors_info fields * explicit colorbar positioning, maintain previous defaults if not set * cbar_ax_left_start_pos * If set, explicitly set the left start position for the colorbar axis, relative to figure * Else if 'cbar_full_width' is set, set to "left_margin" * Else default to 2*left_margin * cbar_ax_bottom_start_pos * If set, explicitly set the bottom start position for the colorbar axis, relative to figure * Else default to 0.05 * cbar_ax_width * If set, explicitly set the width (left to right) of the colorbar axis, relative to figure * Else if 'cbar_full_width' is set, set to right_margin - left_margin * Else default to 1 - 4*left_margin * cbar_ax_height * If set, explicitly set the height (bottom to top) of the colorbar axis, relative to figure * Else, default to 0.02 * explicit colorbar keyword args (mpl_colors_info['colorbar_kwargs']) * colorbar_kwargs['orientation'] * If set, explicitly set orientation * Else, default to 'horizontal' * colorbar_kwargs['extend'] * If set, explicitly set extend option to colorbar call * Else, default to 'both' * colorbar_kwargs['spacing'] * If set, explicitly set 'spacing' option to colorbar call * Else, if 'cbar_spacing' set, use mpl_colors_info['cbar_spacing'] * Else, default to 'proportional' * explicit set_ticks_kwargs args (mpl_colors_info['set_ticks_kwargs']) * set_ticks_kwargs['size'] * If set, explicitly set 'size' option to set_ticks call * Else, default to 'small' * set_ticks_kwargs['labels'] * If set, explicitly set 'labels' option to set_ticks call * Else, default to mpl_colors_info['cbar_tick_labels'] * Else, default to mpl_colors_info['cbar_ticks'] * explicit set_label_kwargs (mpl_colors_info['set_label_kwargs]) * set_label_kwargs['size'] * If set, explicitly set 'size' option to set_label call * Else, default to rc_params['font.size'] * Call pyplot.colorbar vs fig.colorbar * Pass "cbar_kwargs" in directly to allow specifying arbitrary colorbar options via mpl_colors_info * Pass **set_ticks_kwargs to cbar.set_ticks call** * Pass **set_label_kwargs to cbar.set_label call** modified: geoips/interface_modules/output_formats/imagery_clean.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Support plotting on existing figure and axis * Only create fig, main_ax, and mapobj if not passed in explicitly * If fig, main_ax, and mapobj passed, plot on existing * Only output final image if output_fnames is not None modified: geoips/interface_modules/output_formats/imagery_windbarbs.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Support plotting on existing figure and axis * Update output_clean_windbarbs function to take fig, main_ax, and mapobj arguments * Only create figure, main axis, and mapobj if not passed * Only output image file if clean_fnames is not None * Allow specifying barb_sizes in product_definition :: If 'barb_sizes' is in xarray_obj.attrs['product_definition'], use those values * thinning * barb_length * line_width * sizes_dict * rain_size Else, default to former operation based on product_name == 'windbarbs' or 'wind-ambiguities' modified: geoips/interface_modules/output_formats/imagery_windbarbs_clean.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Support plotting on existing figure and axis * Update imagery_windbars_clean function to take fig, main_ax, and mapobj arguments * These are passed directly through to imagery_windbarbs.py output_clean_windbarbs function modified: geoips/interface_modules/procflows/single_source.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Support plotting data without producing output * Add "no_output" option to "plot_data" function - do not produce output files if set, only plot * Support get_area_defs_from_command_line_args with METADATA only available * Make "variables" argument optional - currently unused anyway modified: geoips/interface_modules/user_colormaps/pmw_tb/cmap_Rain.py ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Use colorbar_kwargs and set_ticks_kwargs options for demonstration purposes * Same functionality as previously, just using explicit keyword argument specifications VIIRS reader bug-fix for terminator case ======================================== GEOIPS/geoips#104: 2022-10-21, VIIRS reader bug-fix for terminator case Bug fixes --------- geoips/interface_modules/readers/viirs_netcdf.nc ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Move VIIRS solar reflective bands to neww data_type: * MOD-Vis: M01, M02, M03, M04, M05, M06, M09 * IMG-Vis: I01, I02, I03 * These reflective bands are not present in nighttime granules, and causes issues when dealing with a pair of granules that cross the terminator. * Reader now capable of reading geo fields from a single file into multiple datasets use 'conda-forge' vs 'defaults' =============================== GEOIPS/geoips#98: 2022-09-28, use 'conda-forge' vs 'defaults' Installation ------------ setup.sh: default to '-c conda-forge', allow '-c defaults' by request for conda commands: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * **setup.sh conda_install**: use Miniforge by default, Miniconda if "conda_defaults_channel" passed * **setup.sh conda_update**: Use conda-forge by default, "defaults" if "conda_defaults_channel" passed * **setup.sh create_geoips_conda_env**: Use conda-forge by default, "defaults" if "conda_defaults_channel" passed * **setup.sh install**: matplotlib and cartopy still must use conda-forge * Remove version specifications for matplotlib and cartopy (allow latest until test outputs break) setup.py: Update versions to allow latest, but maintain specifically pre-installed versions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * base: matplotlib>=3.5.3 (CI/CD installation requires 3.5.3 to work with cartopy) * base: shapely>=1.8.2 (CI/CD installation requires specific 1.8.2 build) * base: cartopy>=0.20.3 (CI/CD installation requires 0.20.3 to work with shapely) * test_outputs: matplotlib>=3.6.0 (update outputs to latest) * test_outputs: cartopy>=0.21.0 (update outputs to latest) * cicd_pipeline: Add specific matplotlib (3.5.3), cartopy (0.20.3), and shapely (1.8.2 pre-built) versions * This is NOT called from default interactive installation README.md ^^^^^^^^^ * Update GEOIPS_ACTIVE_BRANCH to dev for NRLONLY * add GEOIPS_PACKAGES_DIR, GEOIPS_TESTDATA_DIR, and GEOIPS_DEPENDENCIES_DIR env vars for completeness * Do not use GEOIPS BASEDIR within README EXCEPT to set above env vars * Pass "conda-forge" to base_install_and_test.sh to explicitly request "conda-forge" channel base_install_and_test.sh ^^^^^^^^^^^^^^^^^^^^^^^^ * Pass $conda_channel to setup.sh commands: conda_install, conda_update, create_geoips_conda_env * Separate update_conda and create_geoips_conda_env steps