# # # Distribution Statement A. Approved for public release. Distribution unlimited.
# # #
# # # Author:
# # # Naval Research Laboratory, Marine Meteorology Division
# # #
# # # This program is free software: you can redistribute it and/or modify it under
# # # the terms of the NRLMMD License included with this program. This program is
# # # distributed WITHOUT ANY WARRANTY; without even the implied warranty of
# # # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the included license
# # # for more details. If you did not receive the license, for more information see:
# # # U-S-NRL-Marine-Meteorology-Division/

Version 1.10.0a12 (2023-05-09)#

Overall YAML interface updates

  • Remove old YAML discovery functions in geoips_utils

  • Remove old style YAML configs

  • Update dev/output_config.py to use new YAML interfaces

  • Rework mapping routines to use new interfaces

Feature and gridline annotators interface

  • Add new feature and gridline annotators YAML-based interface

  • Refactor gridline_annotator and feature_annotator plugin schema and plugins

  • Update boundaries/gridlines_info to feature/gridline_annotator

  • Update boundaries/gridlines_params to feature/gridline_annotator

Sectors interface

  • Add new sector YAML-based interface

  • Remove sectorfiles option from commandline

Dynamic sectors interface

  • Update sector interface to include dynamic sectors

  • Update all TC templates to use updated sectors.generated schema

  • Update command line argument from tc_template_yaml to tc_spec_template

Sector adjusters interface

  • Add sector_adjusters module-based interface

  • Update single_source procflow to use new sector_adjusters interface

  • Replace adjust_area_def command line arg with sector_adjuster

Colormaps interface

  • Add colormap module-based interface

  • Add plugins/**.txt to pyproject.toml

  • Move colormaps from geoips/image_utils/ascii_palettes to plugins/txt/ascii_palettes

  • Update matplotlib_linear_norm to be a “configurable” colormap

    • Add “find_ascii_palette” routine to geoips_utils.py

    • Allow cbar_ticks == None to plot min/max of actual data

    • Allow matplotlib, ascii, and geoips colormap sources

  • Change colormap families to “matplotlib”

  • Run pytest on all installed geoips plugin packages

Products interface

  • Add products YAML-based interface

  • Update procflows to use products interface

  • Allow passing dictionary of product_spec_overrides to products.get_plugin

  • Support multiple coverage checkers for a single product

  • Bug fixes

    • Fix plugin names in product_defaults

  • Testing updates

    • Add data_fusion stitched test to test_full_install.sh

    • Add template_basic_plugin test to test_full_install.sh

Bug fixes#

Fix plugin names in product_defaults#

From NRLMMD-GEOIPS/geoips#211f: 2023-05-09, product updates

  • Update range -> output_data_range in Infared-Gray product_defaults

  • pmw_tc.pmw_ -> pmw_ algorithm name in PMW product_defaults

modified: plugins/yaml/product_defaults/pmw_37/37pct.yaml
modified: plugins/yaml/product_defaults/pmw_37/37pctNearest.yaml
modified: plugins/yaml/product_defaults/pmw_89/color89.yaml
modified: plugins/yaml/product_defaults/pmw_89/color89Nearest.yaml
modified: plugins/yaml/product_defaults/visir/Infrared-Gray.yaml

Major New Functionality#

Add new sector interface#

From NRLMMD-GEOIPS/geoips#211b: 2023-05-01, sector updates

This interface can be used for both dynamic templates and fully specified static yaml-based sectors.

new file: interfaces/yaml_based/sectors.py

Add sector_adjusters interface#

From NRLMMD-GEOIPS/geoips#211d: 2023-05-02, sector adjuster updates

This is only used within the recenter_tc repository, but interface added here.

new: geoips/interfaces/module_based/sector_adjusters.py
modified: geoips/interfaces/__init__.py

Allow cbar_ticks == None to plot min/max of actual data#

From NRLMMD-GEOIPS/geoips#211e: 2023-05-03, colormap updates

Update create_colorbar function to set cbar_ticks to cmap_norm.vmin and cmap_norm.vmax if cbar_ticks is “None”. Additionally, ensure if the “kwargs” options are included in mpl_colors_info, they are only used if not None (allows including defaults for all fields in mpl_colors_info)

modified:   geoips/image_utils/mpl_utils.py

Update matplotlib_linear_norm to allow matplotlib, ascii, and geoips colormaps#

From NRLMMD-GEOIPS/geoips#211e: 2023-05-03, colormap updates

Pass “cmap_source” option to determine where to find the colormap. Default to “matplotlib” (so backwards compatible)

  1. matplotlib builtin

  2. ascii palette

  3. geoips plugin

modified: geoips/plugins/modules/colormaps/matplotlib_linear_norm.py

Add “find_ascii_palette” routine to geoips_utils.py#

From NRLMMD-GEOIPS/geoips#211e: 2023-05-03, colormap updates

This searches the plugin directory for plugins/txt/ascii_palettes (for now). May tune this additionally, but for now we will hard code txt/ascii_palettes, and require ascii colormaps to have extension .txt.

modified: geoips/geoips_utils.py

Update matplotlib_linear_norm colormap plugin to support ASCII palettes#

From NRLMMD-GEOIPS/geoips#211e: 2023-05-03, colormap updates

This plugin provides the ability to fully specify matplotlib color info via call signature arguments. Add options for:

  • cmap_name - name of colormap to use

  • cmap_source - one of “matplotlib”, “geoips”, or “ascii”

  • cmap_path - optional full path to ascii palette

  • cbar_tick_labels - alternative labels to use if not the numeric values of cbar_ticks

  • cbar_spacing - cbar spacing, proportional or uniform

  • cbar_full_width - specify colorbar should be the full width of image

  • colorbar_kwargs - pass through to matplotlib “colorbar” command

  • set_ticks_kwargs - pass through to “set_ticks” command

Also add these fields to the colormap YAML schema.

Additionally, update image_utils.colormap_utils.from_ascii to take optional “cmap_name” kwarg, if not specified, just use the basename of fname.

modified: geoips/schema/product_defaults/bases/colormap.yaml
geoips/image_utils/colormap_utils.py
geoips/plugins/modules/colormaps/matplotlib_linear_norm.py

Breaking Changes#

Remove old YAML discovery functions in geoips_utils#

From NRLMMD-GEOIPS/geoips#211a: 2023-05-01, gridline/feature updates

  • Remove list_gridlines_params_dict_yamls

  • Remove list_boundaries_params_dict_yamls

geoips/geoips_utils.py

Remove old YAML configs and old interfaces#

From NRLMMD-GEOIPS/geoips#211: 2023-05-01, update code for new interfaces

NOTE: had to pull “set_lonlat_spacing” out of geoips/dev/gridlines.py and put it in geoips/dev/output_config.py. Will likely want to handle this differently in the future.

  • boundaries -> feature_annotator

  • gridlines -> gridline_annotator

  • sectors_static -> sectors.static

  • sectors_dynamic -> sectors.dynamic

  • product_inputs -> products

  • product_params -> product_defaults

modified: geoips/dev/output_config.py
deleted: geoips/dev/boundaries.py
deleted: geoips/dev/gridlines.py
deleted: geoips/yaml_configs/plotting_params/boundaries
deleted: geoips/yaml_configs/plotting_params/gridlines
deleted: geoips/yaml_configs/sectors_static
deleted: geoips/yaml_configs/sectors_dynamic
deleted: geoips/yaml_configs/product_inputs
deleted: geoips/yaml_configs/product_params

Update dev/output_config.py to use new YAML interfaces#

From NRLMMD-GEOIPS/geoips#211: 2023-05-01, update code for new interfaces

  • from geoips.dev.boundaries import get_boundaries

  • from geoips.dev.gridlines import get_gridlines

to:

  • from geoips.interfaces import feature_annotators

  • from geoips.interfaces import gridline_annotators

modified: geoips/dev/output_config.py

Refactor gridline_annotator and feature_annotator plugin schema and plugins#

From NRLMMD-GEOIPS/geoips#211a: 2023-05-01, gridline/feature updates

  • In feature annotators

    • use “coastlines” -> “coastline” and “countries” -> “borders”

    • use “edgecolor” instead of “color”

  • In gridline annotators

    • Move to three spec sections: “spacing”, “labels”, and “lines”

    • Use same line style for meridians and parallels

::

modified: schema/feature_annotators/cartopy.yaml modified: schema/gridline_annotators/cartopy.yaml modified: plugins/yaml/feature_annotators modified: plugins/yaml/gridline_annotators modified: tests/test_pytest/test_plugin_schema/bad/feature_annotators/cartopy.yaml modified: tests/test_pytest/test_plugin_schema/bad/gridline_annotators/cartopy.yaml

Rework mapping routines to use new interfaces#

From NRLMMD-GEOIPS/geoips#211a: 2023-05-01, gridline/feature updates

  • image_utils/maps.py

    • Remove all functionality relating to Basemap

    • Remove set_boundaries_info_dict and set_gridlines_info_dict

    • Rename check_gridlines_info_dict to check_gridline_annotator

    • Rename check_boundaries_info_dict to check_feature_annotator

    • Add compute_lat_auto_spacing and compute_lon_auto_spacing

    • Refactor all other functions

  • image_utils/mpl_utils.py

    • Replace all uses of boundaries_info with feature_annotator

    • Replace all uses of gridlines_info with gridline_annotator

::

image_utils/maps.py image_utils/mpl_utils.py

Update boundaries/gridlines_info to feature/gridline_annotator#

From NRLMMD-GEOIPS/geoips#211a: 2023-05-01, gridline/feature updates

These are dictionaries of feature information to use in the final output plot. Replace with “feature_annotator” terminology for the call signature argument.

modified: geoips/dev/output_config.py
modified: geoips/interfaces/module_based/output_formatters.py
modified: geoips/plugins/modules/output_formatters/full_disk_image.py
modified: geoips/plugins/modules/output_formatters/imagery_annotated.py
modified: geoips/plugins/modules/output_formatters/imagery_windbarbs.py
modified: geoips/image_utils/mpl_utils.py

Update boundaries/gridlines_params to feature/gridline_annotator#

From NRLMMD-GEOIPS/geoips#211a: 2023-05-01, gridline/feature updates

Update both command line calls and YAML output configs.

modified: geoips/commandline/args.py
modified: geoips/dev/output_config.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: tests/scripts/amsr2.tc.89H-Physical.imagery_annotated.sh
modified: tests/scripts/amsub_mirs.tc.183-3H.imagery_annotated.sh
modified: tests/scripts/hy2.tc.windspeed.imagery_annotated.sh
modified: tests/scripts/saphir.tc.183-3HNearest.imagery_annotated.sh
modified: tests/scripts/sar.tc.nrcs.imagery_annotated.sh
modified: tests/scripts/viirsday.tc.Night-Vis-IR.imagery_annotated.sh
modified: tests/yaml_configs/abi_test.yaml
modified: tests/yaml_configs/abi_test_low_memory.yaml

Remove sectorfiles option from commandline#

From NRLMMD-GEOIPS/geoips#211b: 2023-05-01, sector updates

Sectors are now fully fledged plugins, so we can just pass the list of sectors command line, we do not have to pass the full path to the associated file. Remove all references to “sectorfiles” arguments, and rely on the sector list.

modified: geoips/commandline/args.py
modified: geoips/commandline/run_procflow.py
modified: geoips/dev/output_config.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: geoips/sector_utils/utils.py
modified: tests/scripts/abi.static.Infrared.imagery_annotated.sh
modified: tests/scripts/abi.static.Visible.imagery_annotated.sh
modified: tests/scripts/documentation_imagery.sh
modified: tests/scripts/ewsg.static.Infrared.imagery_clean.sh
modified: tests/scripts/mimic_coarse.static.TPW-CIMSS.imagery_annotated.sh
modified: tests/yaml_configs/abi_test.yaml
modified: tests/yaml_configs/abi_test_low_memory.yaml
modified: tests/yaml_configs/amsr2_test.yaml
modified: tests/yaml_configs/amsr2_test_low_memory.yaml

Update all TC templates to use updated sectors.generated formatting#

From NRLMMD-GEOIPS/geoips#211c: 2023-05-02, dynamic sector updates

These only require sector_spec_generator specifications, not sector_metadata_generator. Metadata is determined separately from the area definition.

Update schema and all TC template YAMLs accordingly.

modified: geoips/schema/sectors/generated.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_1024x1024/tc_1km_1024x1024.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_1024x1024/tc_2km_1024x1024.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_1400x1400/tc_1km_1400x1400.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_1400x1400/tc_2km_1400x1400.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_1600x1600/tc_2km_1600x1600.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_1600x1600/tc_4km_1600x1600.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_256x256/tc_4km_256x256.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_512x512/tc_2km_512x512.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_512x512/tc_4km_512x512.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_800x800/tc_2km_800x800.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_800x800/tc_4km_800x800.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_huge/tc_0p1km_3200x3200.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_huge/tc_1km_2500x2500.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_huge/tc_1km_3200x3200.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_web_ascat_high_barbs_template.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_web_ascatuhr_barbs_template.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_web_halfkm_template.yaml
modified: geoips/plugins/yaml/sectors/dynamic/tc_web_template.yaml

Replace tc_template_yaml command line argument with tc_spec_template#

From NRLMMD-GEOIPS/geoips#211c: 2023-05-02, dynamic sector updates

Now that TC templates are fully fledged plugins, reference them just based on plugin name rather than full path to YAML file.

modified: geoips/commandline/args.py
modified: geoips/dev/output_config.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: geoips/sector_utils/tc_tracks.py
modified: scripts/ascat_knmi.tc.windbarbs.imagery_windbarbs_clean.sh
modified: scripts/ascat_uhr.tc.wind-ambiguities.imagery_windbarbs.sh
modified: scripts/atms.tc.165H.netcdf_geoips.sh
modified: scripts/oscat_knmi.tc.windbarbs.imagery_windbarbs.sh
modified: yaml_configs/abi_test.yaml
modified: yaml_configs/abi_test_low_memory.yaml
modified: yaml_configs/amsr2_test.yaml
modified: yaml_configs/amsr2_test_low_memory.yaml

Update single_source procflow to use new sector_adjusters interface#

From NRLMMD-GEOIPS/geoips#211d: 2023-05-02, sector adjuster updates

Was previously using “find_entry_point” directly.

modified:   plugins/modules/procflows/single_source.py

Replace adjust_area_def command line arg with sector_adjuster#

From NRLMMD-GEOIPS/geoips#211d: 2023-05-02, sector adjuster updates

Update appropriately in config_based and single_source.

modified: commandline/args.py
modified: plugins/modules/procflows/config_based.py
modified: plugins/modules/procflows/single_source.py

Move colormaps to plugins/txt/ascii_palettes#

From NRLMMD-GEOIPS/geoips#211e: 2023-05-03, colormap updates

Consolidating user-defined capabilities within “plugins” directory.

Add support for text ascii_palettes - which are not strictly geoips plugins (since they will not have the required attributes, etc), but will be accessed in a similar way.

modified: pyproject.toml
modified: geoips/plugins/modules/colormaps/tpw/tpw_cimss.py
modified: geoips/plugins/modules/colormaps/tpw/tpw_purple.py
modified: geoips/plugins/modules/colormaps/tpw/tpw_pwat.py
renamed: geoips/image_utils/ascii_palettes/tpw_cimss.txt -> geoips/plugins/txt/ascii_palettes/tpw_cimss.txt
renamed: geoips/image_utils/ascii_palettes/tpw_purple.txt -> geoips/plugins/txt/ascii_palettes/tpw_purple.txt
renamed: geoips/image_utils/ascii_palettes/tpw_pwat.txt -> geoips/plugins/txt/ascii_palettes/tpw_pwat.txt

Change all colormap families to “matplotlib”#

From NRLMMD-GEOIPS/geoips#211e: 2023-05-03, colormap updates

Redefining plugin familes as a collection of required_parameters, required_kwargs, and allowable_kwargs. This allows us to have a collections of keyword arguments that can be specified for matplotlib-based colormaps, but do not have to be specified. So rather than having a different family for every combination of tuning parameters for color specifications, have a general “matplotlib” family that all return the “mpl_colors_info” dictionary, but can have a variable set of arguments (depending on requirements for a specific colormap).

All matplotlib families now have NO required params or kwargs, and a list of available_kwargs.

modified: geoips/interfaces/module_based/colormaps.py
modified: geoips/plugins/modules/colormaps/cmap_rgb.py
modified: geoips/plugins/modules/colormaps/matplotlib_linear_norm.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_150H.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_37H.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_37H_Legacy.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_37H_Physical.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_37pct.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_89H.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_89HW.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_89H_Legacy.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_89H_Physical.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_89pct.py
modified: geoips/plugins/modules/colormaps/pmw_tb/cmap_Rain.py
modified: geoips/plugins/modules/colormaps/tpw/tpw_cimss.py
modified: geoips/plugins/modules/colormaps/tpw/tpw_purple.py
modified: geoips/plugins/modules/colormaps/tpw/tpw_pwat.py
modified: geoips/plugins/modules/colormaps/visir/IR_BD.py
modified: geoips/plugins/modules/colormaps/visir/Infrared.py
modified: geoips/plugins/modules/colormaps/visir/WV.py
modified: geoips/plugins/modules/colormaps/winds/wind_radii_transitions.py

Allow passing dictionary of product_spec_overrides to products.get_plugin#

From NRLMMD-GEOIPS/geoips#211f: 2023-05-09, product updates

When opening a product plugin, if optional “product_spec_override” is passed to the get_plugin method, use the values contained within product_spec_override in place of those found in products and product_defaults.

This replaces the “product_params_override” used with the deprecated product interface.

modified: geoips/commandline/args.py
modified: geoips/interfaces/yaml_based/products.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: tests/scripts/amsr2.tc.89H-Physical.imagery_annotated.sh
modified: tests/scripts/amsr2_ocean.tc.windspeed.imagery_clean.sh
modified: tests/scripts/amsub_mirs.tc.183-3H.imagery_annotated.sh
modified: tests/scripts/ascat_knmi.tc.windbarbs.imagery_windbarbs_clean.sh
modified: tests/scripts/ascat_low_knmi.tc.windbarbs.imagery_windbarbs.sh
modified: tests/scripts/ascat_uhr.tc.wind-ambiguities.imagery_windbarbs.sh
modified: tests/scripts/atms.tc.165H.netcdf_geoips.sh
modified: tests/scripts/gmi.tc.89pct.imagery_clean.sh
modified: tests/scripts/hy2.tc.windspeed.imagery_annotated.sh
modified: tests/scripts/oscat_knmi.tc.windbarbs.imagery_windbarbs.sh
modified: tests/scripts/saphir.tc.183-3HNearest.imagery_annotated.sh
modified: tests/scripts/sar.tc.nrcs.imagery_annotated.sh
modified: tests/scripts/ssmi.tc.37pct.imagery_clean.sh
modified: tests/scripts/viirsday.tc.Night-Vis-IR.imagery_annotated.sh
modified: tests/yaml_configs/amsr2_test.yaml
modified: tests/yaml_configs/amsr2_test_low_memory.yaml

Support multiple coverage checkers for a single product#

From NRLMMD-GEOIPS/geoips#211f: 2023-05-09, product updates

  • Add filename_coverage_checker, full_coverage_checker, and image_production_coverage_checker to product_defaults schema

  • Pass output_dict[“product_spec_override”] to all products.get_plugin calls.

modified: geoips/plugins/modules/procflows/config_based.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: geoips/dev/output_config.py
modified: geoips/plugins/modules/filename_formatters/utils/tc_file_naming.py
modified: geoips/plugins/modules/output_formatters/imagery_annotated.py
modified: geoips/plugins/modules/output_formatters/metadata_tc.py
modified: geoips/schema/product_defaults/specs/algorithm_colormap.yaml
modified: geoips/schema/product_defaults/specs/algorithm_interpolator_colormap.yaml
modified: geoips/schema/product_defaults/specs/interpolator_algorithm.yaml
modified: geoips/schema/product_defaults/specs/interpolator_algorithm_colormap.yaml

Testing Updates#

Run pytest on all installed geoips plugin packages#

From NRLMMD-GEOIPS/geoips#211: 2023-05-09, interface updates

To ensure all functionality is tested, loop through all geoips plugin packages when running tests/test_pytest.

modified: tests/test_pytest/test_all_yaml_plugins.py

Add data_fusion stitched test to test_full_install.sh#

From NRLMMD-GEOIPS/geoips#211: 2023-05-09, interface updates

Add NOAA AWS pull of goes16, goes17, and himawari8 to test_data_fusion.

Update data_fusion test call to run test_all.sh vs just layered.sh

modified:   setup.sh
modified:   tests/test_full_install.sh

Add template_basic_plugin test to test_full_install.sh#

From NRLMMD-GEOIPS/geoips#211: 2023-05-09, interface updates

Ensure template_basic_plugin is installed properly.

modified; tests/test_full_install.sh