# # # Distribution Statement A. Approved for public release. Distribution unlimited.
# # #
# # # Author:
# # # Naval Research Laboratory, Marine Meteorology Division
# # #
# # # This program is free software: you can redistribute it and/or modify it under
# # # the terms of the NRLMMD License included with this program. This program is
# # # distributed WITHOUT ANY WARRANTY; without even the implied warranty of
# # # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the included license
# # # for more details. If you did not receive the license, for more information see:
# # # U-S-NRL-Marine-Meteorology-Division/

Version 1.5.3 (2022-11-07)#

v1.5.3: 2022-11-07, update python install process, alternate command line args funcs, unique storm dirnames for invests, bug fixes

2022-11-04 bug fixes#

GEOIPS/geoips#108: 2022-11-04, bug fixes

Documentation Updates#

  • Clarify steps in updating CHANGELOG during internal release process

modified: CHANGELOG_TEMPLATE.md

Test Repo Update#

  • Update abi.tc.Infrared.imagery_annotated metadata YAML

    • Add Add storm_start_datetime to metadata yaml

modified:   tests/outputs/abi.tc.Infrared.imagery_annotated/20200918_195020_AL202020_abi_goes-16_Infrared_110kts_100p00_1p0.png.yaml

fix sun.set_time bug in overpass_predictor.py#

GEOIPS/geoips#114: 2022-10-27, fix sun.set_time bug in overpass_predictor.py

Bug fixes#

Fix sun.set_time bug in overpass_predictor.py#

  • Previously only check sun.rise_time

  • sun.rise_time can be valid while sun.set_time is None, so must check both.

  • return None if either sun.rise_time or sun.set_time is None:

    modified:   geoips/sector_utils/overpass_predictor.py

VIIRS lat/lon variables may contain unmasked fill values#

GEOIPS/geoips#113: 2022-10-27, VIIRS lat/lon variables may contain unmasked fill values

Bug fix#

modified: geoips/interface_modules/readers/viirs_netcdf.py#

  • Loop through xarray_return data types, and mask latitude/longitude by their fill value

cowvr edge case - no usable sectored data#

GEOIPS/geoips#111: 2022-10-27, cowvr edge case - no usable sectored data

Bug fix#

modified: geoips/interface_modules/procflows/config_based.py#

  • Add check if sectored data contains any usable data

    • Only add variable to all_vars list if sectored data set is not entirely comprised of NaNs

modified: geoips/interface_modules/procflows/single_source.py#

  • Add check if sectored data contains any usable data

    • Only add variable to all_vars list if sectored data set is not entirely comprised of NaNs

add sectored and unsectored AMSR2 winds outputs#

GEOIPS/geoips_system_nrl#61: 2022-10-27, add sectored and unsectored AMSR2 winds outputs

Major New Functionality#

Add sectored and unsectored products to product_inputs/amsr2.yaml#

  • Allows producing sectored and unsectored text winds outputs:

    modified: geoips/yaml_configs/product_inputs/amsr2.yaml

get_product allow empty dict, bug fixes#

GEOIPS/data_fusion#8: 2022-10-21, get_product allow empty dict, bug fixes

modified: geoips/dev/product.py#

  • Previously would only replace individual fields within dictionaries in product_inputs

  • Now, if an empty dictionary is included within product_inputs, it will replace the entire product_params

    dictionary with an empty dictionary.

  • This is useful when using product_templates, and replacing the algorithm or colormap, etc

Test Repo Updates#

Update ASCAT UHR test data path#

  • Renamed ASCAT UHR test data subdirectories with additional test cases:

    modified: tests/scripts/ascat_uhr.tc.wind-ambiguities.imagery_windbarbs.sh

Turn off minor ticks#

  • matplotlib 3.6.0 sometimes has inconsistent results with including minor ticks or not.

  • Unclear why it impacts some colorbars and not others.

  • We may eventually add support for including minor ticks within mpl_colors_info, but for now

  • explicitly turn off minor ticks so outputs will continue to match (use the old default):

    modified: geoips/image_utils/mpl_utils.py

create unique storm dirnames for invests#

GEOIPS/geoips#103: 2022-10-17, create unique storm dirnames for invests

Major New Functionality#

geoips/interface_modules/filename_formats/tc_clean_fname.py#

  • Allow passing “output_dict” to allow using unique directory name for INVESTS

geoips/interface_modules/filename_formats/tc_fname.py#

  • def tc_fname

    • Allow passing output_dict to provide current output parameters for overall filename specifications

  • def assemble_tc_fname

    • Allow passing both “output_dict” and “sector_info” to allow timestamp in dirname for INVESTS

geoips/interface_modules/filename_formats/utils/tc_file_naming.py#

  • Allow passing both “output_dict” and “sector_info” to support .%Y%m%d%H dirname for INVESTS

    • output_dict[‘file_path_modifications’][‘unique_invest_dirs’] True

    • storm_start_datetime is datetime object

      • sector_info[‘original_storm_start_datetime’] if it exists, else

      • sector_info[‘storm_start_datetime’]

    • storm number > 69 (ie, invest)

    • output_dict[‘file_path_modifications’][‘existing_invest_dirs_allowable_time_diff’] > 0

      • If specified, use existing directory closest in time to storm_start_datetime

      • If none exist, use storm_start_datetime appended to INVEST directory

      • Ie, SH932020.2020020506 vs SH932020

      • If SH932020.2020020406 exists, would use that rather than creating 2020020506

      • SH162020 does NOT contain the extra storm start datetime information

geoips/interface_modules/trackfile_parsers/bdeck_parser.py#

  • Add storm_start_datetime field to bdeck sector info

    • pull from first entry in bdeck file

  • Add original_storm_start_datetime field to bdeck sector_info

    • Pull from filename if available (since bdeck entries can change)

    • DO NOT INCLUDE in dictionary if it is not available

    • If it exists, this will be a more consistent value than storm_start_datetime (which can change with subsequent deck files)

geoips/interface_modules/filename_formats/metadata_default_fname.py#

  • def metadata_default_fname

    • Allow passing output_dict to provide current output parameters for overall filename specifications

  • def assemble_metadata_default_fname

    • Allow passing both “output_dict” and “sector_info” to allow timestamp in dirname for INVESTS

geoips/interface_modules/filename_formats/text_winds_tc_fname.py#

  • def text_winds_tc_fname

    • Allow passing output_dict to provide current output parameters for overall filename specifications

  • def assemble_text_winds_tc_fname

    • Allow passing both “output_dict” and “sector_info” to allow timestamp in dirname for INVESTS

Test Repo Updates#

Add storm_start_datetime to YAML metadata outputs#

  • modified: tests/outputs/abi.tc.IR-BD.imagery_annotated/20200918_195020_AL202020_abi_goes-16_IR-BD_110kts_100p00_1p0.png.yaml

  • modified: tests/outputs/abi.tc.Visible.imagery_annotated/20200918_195020_AL202020_abi_goes-16_Visible_110kts_100p00_1p0.png.yaml

  • modified: tests/outputs/amsr2.tc.89H-Physical.imagery_annotated/20200518_073601_IO012020_amsr2_gcom-w1_89H-Physical_140kts_100p00_res1p0-cr300.png.yaml

  • modified: tests/outputs/amsub_mirs.tc.183-3H.imagery_annotated/20210419_235400_WP022021_amsu-b_metop-a_183-3H_115kts_100p00_1p0.png.yaml

  • modified: tests/outputs/ascat_knmi.tc.windbarbs.imagery_windbarbs_clean/20210421_014248_WP022021_ascat_metop-c_windbarbs_120kts_78p20_0p5-clean.png.yaml

  • modified: tests/outputs/ascat_low_knmi.tc.windbarbs.imagery_windbarbs/20210421_014156_WP022021_ascat_metop-c_windbarbs_120kts_35p17_1p0.png.yaml

  • modified: tests/outputs/ascat_uhr.tc.wind-ambiguities.imagery_windbarbs/20210421_014200_WP022021_ascatuhr_metop-c_wind-ambiguities_120kts_100p00_0p1.png.yaml

  • modified: tests/outputs/gmi.tc.89pct.imagery_clean/20200917_172045_AL202020_gmi_GPM_89pct_115kts_78p16_res1p0-cr300-clean.png.yaml

  • modified: tests/outputs/hy2.tc.windspeed.imagery_annotated/20211202_084039_WP272021_hscat_hy-2b_windspeed_95kts_97p06_1p0.png.yaml

  • modified: tests/outputs/mimic_fine.tc.TPW-PWAT.imagery_annotated/20210419_230000_WP022021_mimic_tpw_TPW-PWAT_115kts_100p00_1p0.png.yaml

  • modified: tests/outputs/oscat_knmi.tc.windbarbs.imagery_windbarbs/20210209_025351_SH192021_oscat_scatsat-1_windbarbs_135kts_75p10_1p0.png.yaml

  • modified: tests/outputs/saphir.tc.183-3HNearest.imagery_annotated/20210209_003103_SH192021_saphir_meghatropiques_183-3HNearest_135kts_88p76_1p0.png.yaml

  • modified: tests/outputs/sar.tc.nrcs.imagery_annotated/20181025_203206_WP312018_sar-spd_sentinel-1_nrcs_130kts_58p51_res1p0-cr300.png.yaml

  • modified: tests/outputs/ssmi.tc.37pct.imagery_clean/20200519_080900_IO012020_ssmi_F15_37pct_110kts_50p65_1p0-clean.png.yaml

  • modified: tests/outputs/viirsday.tc.Night-Vis-IR.imagery_annotated/20210209_074210_SH192021_viirs_noaa-20_Night-Vis-IR_130kts_100p00_1p0.png.yaml

Bug fixes#

Do not attempt to set_ticks if cbar_ticks is not defined geoips/image_utils/mpl_utils.py#

Replace fig.savefig frameon=False argument with facecolor=”none”#

allow alternate command line args funcs#

GEOIPS/data_fusion#8: 2022-09-29, allow alternate command line args funcs

Enhancements#

new: tests/sectors/tc_bdecks/bwp142022.dat#

modified: geoips/commandline/args.py#

  • Allow passing alternate check_args_func and get_args_func to get_command_line_args

  • Default output_format None vs imagery_annotated

modified: geoips/commandline/run_procflow.py#

  • Allow passing alternate “get_command_line_args” func to run_procflow main

modified: geoips/dev/product.py#

  • Add ‘xarray_dict_to_output_format’ product type

  • Allow specifying “product_template” within product_params YAML as well as product_inputs

modified: geoips/image_utils/mpl_utils.py#

  • Support additional mpl_colors_info fields

    • explicit colorbar positioning, maintain previous defaults if not set

      • cbar_ax_left_start_pos

        • If set, explicitly set the left start position for the colorbar axis, relative to figure

        • Else if ‘cbar_full_width’ is set, set to “left_margin”

        • Else default to 2*left_margin

      • cbar_ax_bottom_start_pos

        • If set, explicitly set the bottom start position for the colorbar axis, relative to figure

        • Else default to 0.05

      • cbar_ax_width

        • If set, explicitly set the width (left to right) of the colorbar axis, relative to figure

        • Else if ‘cbar_full_width’ is set, set to right_margin - left_margin

        • Else default to 1 - 4*left_margin

      • cbar_ax_height

        • If set, explicitly set the height (bottom to top) of the colorbar axis, relative to figure

        • Else, default to 0.02

    • explicit colorbar keyword args (mpl_colors_info[‘colorbar_kwargs’])

      • colorbar_kwargs[‘orientation’]

        • If set, explicitly set orientation

        • Else, default to ‘horizontal’

      • colorbar_kwargs[‘extend’]

        • If set, explicitly set extend option to colorbar call

        • Else, default to ‘both’

      • colorbar_kwargs[‘spacing’]

        • If set, explicitly set ‘spacing’ option to colorbar call

        • Else, if ‘cbar_spacing’ set, use mpl_colors_info[‘cbar_spacing’]

        • Else, default to ‘proportional’

    • explicit set_ticks_kwargs args (mpl_colors_info[‘set_ticks_kwargs’])

      • set_ticks_kwargs[‘size’]

        • If set, explicitly set ‘size’ option to set_ticks call

        • Else, default to ‘small’

      • set_ticks_kwargs[‘labels’]

        • If set, explicitly set ‘labels’ option to set_ticks call

        • Else, default to mpl_colors_info[‘cbar_tick_labels’]

        • Else, default to mpl_colors_info[‘cbar_ticks’]

    • explicit set_label_kwargs (mpl_colors_info[‘set_label_kwargs])

      • set_label_kwargs[‘size’]

        • If set, explicitly set ‘size’ option to set_label call

        • Else, default to rc_params[‘font.size’]

  • Call pyplot.colorbar vs fig.colorbar

    • Pass “cbar_kwargs” in directly to allow specifying arbitrary colorbar options via mpl_colors_info

    • Pass set_ticks_kwargs to cbar.set_ticks call

    • Pass set_label_kwargs to cbar.set_label call

modified: geoips/interface_modules/output_formats/imagery_clean.py#

  • Support plotting on existing figure and axis

    • Only create fig, main_ax, and mapobj if not passed in explicitly

    • If fig, main_ax, and mapobj passed, plot on existing

    • Only output final image if output_fnames is not None

modified: geoips/interface_modules/output_formats/imagery_windbarbs.py#

  • Support plotting on existing figure and axis

    • Update output_clean_windbarbs function to take fig, main_ax, and mapobj arguments

    • Only create figure, main axis, and mapobj if not passed

    • Only output image file if clean_fnames is not None

  • Allow specifying barb_sizes in product_definition

If 'barb_sizes' is in xarray_obj.attrs['product_definition'], use those values
    * thinning
    * barb_length
    * line_width
    * sizes_dict
    * rain_size
Else, default to former operation based on product_name == 'windbarbs' or 'wind-ambiguities'

modified: geoips/interface_modules/output_formats/imagery_windbarbs_clean.py#

  • Support plotting on existing figure and axis

    • Update imagery_windbars_clean function to take fig, main_ax, and mapobj arguments

    • These are passed directly through to imagery_windbarbs.py output_clean_windbarbs function

modified: geoips/interface_modules/procflows/single_source.py#

  • Support plotting data without producing output

    • Add “no_output” option to “plot_data” function - do not produce output files if set, only plot

  • Support get_area_defs_from_command_line_args with METADATA only available

    • Make “variables” argument optional - currently unused anyway

modified: geoips/interface_modules/user_colormaps/pmw_tb/cmap_Rain.py#

  • Use colorbar_kwargs and set_ticks_kwargs options for demonstration purposes

    • Same functionality as previously, just using explicit keyword argument specifications

VIIRS reader bug-fix for terminator case#

GEOIPS/geoips#104: 2022-10-21, VIIRS reader bug-fix for terminator case

Bug fixes#

geoips/interface_modules/readers/viirs_netcdf.nc#

  • Move VIIRS solar reflective bands to neww data_type:

    • MOD-Vis: M01, M02, M03, M04, M05, M06, M09

    • IMG-Vis: I01, I02, I03

    • These reflective bands are not present in nighttime granules,

      and causes issues when dealing with a pair of granules that cross the terminator.

  • Reader now capable of reading geo fields from a single file into multiple datasets

use ‘conda-forge’ vs ‘defaults’#

GEOIPS/geoips#98: 2022-09-28, use ‘conda-forge’ vs ‘defaults’

Installation#

setup.sh: default to ‘-c conda-forge’, allow ‘-c defaults’ by request for conda commands:#

  • setup.sh conda_install: use Miniforge by default, Miniconda if “conda_defaults_channel” passed

  • setup.sh conda_update: Use conda-forge by default, “defaults” if “conda_defaults_channel” passed

  • setup.sh create_geoips_conda_env: Use conda-forge by default, “defaults” if “conda_defaults_channel” passed

  • setup.sh install: matplotlib and cartopy still must use conda-forge

    • Remove version specifications for matplotlib and cartopy (allow latest until test outputs break)

setup.py: Update versions to allow latest, but maintain specifically pre-installed versions#

  • base: matplotlib>=3.5.3 (CI/CD installation requires 3.5.3 to work with cartopy)

  • base: shapely>=1.8.2 (CI/CD installation requires specific 1.8.2 build)

  • base: cartopy>=0.20.3 (CI/CD installation requires 0.20.3 to work with shapely)

  • test_outputs: matplotlib>=3.6.0 (update outputs to latest)

  • test_outputs: cartopy>=0.21.0 (update outputs to latest)

  • cicd_pipeline: Add specific matplotlib (3.5.3), cartopy (0.20.3), and shapely (1.8.2 pre-built) versions

    • This is NOT called from default interactive installation

README.md#

  • Update GEOIPS_ACTIVE_BRANCH to dev for NRLONLY

  • add GEOIPS_PACKAGES_DIR, GEOIPS_TESTDATA_DIR, and GEOIPS_DEPENDENCIES_DIR env vars for completeness

    • Do not use GEOIPS BASEDIR within README EXCEPT to set above env vars

  • Pass “conda-forge” to base_install_and_test.sh to explicitly request “conda-forge” channel

base_install_and_test.sh#

  • Pass $conda_channel to setup.sh commands: conda_install, conda_update, create_geoips_conda_env

  • Separate update_conda and create_geoips_conda_env steps