# # # Distribution Statement A. Approved for public release. Distribution unlimited.
# # #
# # # Author:
# # # Naval Research Laboratory, Marine Meteorology Division
# # #
# # # This program is free software: you can redistribute it and/or modify it under
# # # the terms of the NRLMMD License included with this program. This program is
# # # distributed WITHOUT ANY WARRANTY; without even the implied warranty of
# # # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the included license
# # # for more details. If you did not receive the license, for more information see:
# # # U-S-NRL-Marine-Meteorology-Division/

Version 1.10.0a0 (2023-04-04)#

First attempt at YAML-based plugin interfaces. These updates will be superseded by later alpha releases of v1.10.0. Tagging this version for reference.

  • Improve output from estimate area extent script

  • Add YAML-based plugin validator

    • Update plugin validator to follow references and add a base schema

    • Update plugin validator to use interface.family or interface

  • Add new plugin schema

    • “boundaries” plugin schema

    • “gridlines” plugin schema

    • “product_templates” plugin schema (for each of 11 families)

    • “sectors_dynamic” plugin schema

    • Update boundaries and gridlines schema to better handle defaults

  • Convert old YAML to new formats

    • boundaries YAML to new format

    • gridlines YAML to new format

    • product_params YAML to new product_templates format

    • sectors_dynamic YAML to new format

  • Add unit tests to validate yaml-based plugins and schema

Major New Functionality#

Add a validator for YAML-based plugins.#

From issue NRLMMD-GEOIPS/geoips#132: 2023-04-04, Add validator for YAML plugins.

created: /geoips/schema/__init__.py
created: /geoips/schema/plugin_validator.py

Add a schema for validating Gridlines plugins.#

From issue NRLMMD-GEOIPS/geoips#109: 2023-04-04, Add schema for Gridlines plugins.

modified: /geoips/schema/gridlines.yaml

Add a schema for validating Boundaries plugins.#

From issue NRLMMD-GEOIPS/geoips#108: 2023-04-04, Add schema for `Boundaries` plugins.

modified: /geoips/schema/boundaries.yaml

Add schema for validating sectors_dynamic plugins#

From issue NRLMMD-GEOIPS/geoips#114: 2023-04-07, Add `sectors_dynamic` plugins

new file: geoips/schema/sectors_dynamic.yaml

Add schema for validating product_templates plugins#

From issue NRLMMD-GEOIPS/geoips#122: 2023-04-04, Add `product_templates` plugins

product_templates plugins require family-specific schema - each family within the interface has a unique schema for validation.

These schema are added to the “product_templates” subdirectory under geoips/schema

new: geoips/interface_modules/product_templates/alg_cmap.yaml
new: geoips/interface_modules/product_templates/alg_interp_cmap.yaml
new: geoips/interface_modules/product_templates/alg.yaml
new: geoips/interface_modules/product_templates/interp_alg_cmap.yaml
new: geoips/interface_modules/product_templates/interp_alg.yaml
new: geoips/interface_modules/product_templates/interp.yaml

Note the following families have “additionalProprerties: true” set on the spec, which means the actual contents of the products are not validated at all.

We will likely want to update these in the future to be named “unvalidated_*” to make it clear that they are intended to be used as temporary product families during development, and once the format of the family is fully identified, we can define a new family to comply.

new: geoips/interface_modules/product_templates/sectored_xarray_dict_to_output_format.yaml
new: geoips/interface_modules/product_templates/unmodified.yaml
new: geoips/interface_modules/product_templates/unsectored_xarray_dict_area_to_output_format.yaml
new: geoips/interface_modules/product_templates/unsectored_xarray_dict_to_output_format.yaml
new: geoips/interface_modules/product_templates/xarray_dict_to_output_format.yaml

Bug Fixes#

Don’t rely on environment variables in unit tests#

From issue NRLMMD-GEOIPS/geoips#156: 2023-04-06, Don’t rely on environment variables in unit tests

  • Switched from using GEOIPS_PACKAGES_DIR to importlib.resources.files

modified: tests/test_pytest/test_yaml_plugins.py

Enhancements#

Update output from estimate area extent script.#

From issue NRLMMD-GEOIPS/geoips#123: 2023-04-06, update estimate_area_extent

  • estimate_area_extent script now provides copy/pastable formatted yaml

  • Change sector_type to sector_family

modified: geoips/sector_utils/estimate_area_extent.py

Update validator to use both interface and family names#

From issue NRLMMD-GEOIPS/geoips#122: 2023-04-04, Add `product_templates` plugins

This parses the path to the schema file in order to determine the interface and family name.

geoips/schema/<interface>.yaml for interface specific schema. geoips/schema/<interface>/<family>.yaml for family-specific schema.

modified: /geoips/schema/plugin_validator.py

Update plugin validator to follow references and add a base schema for all schema#

From issue NRLMMD-GEOIPS/geoips#155: 2023-04-06, Follow references in plugin validator

modified: /geoips/schema/plugin_validator.py
modified: /geoips/schema/base.yaml
modified: /geoips/schema/gridlines.yaml
modified: /geoips/schema/boundaries.yaml

Update boundaries and gridlines schema to simplify and handle default values better#

From issue NRLMMD-GEOIPS/geoips#144: 2023-04-05, Update boundaries/gridlines schema

modified: /geoips/interface_modules/boundaries/default.yaml
modified: /geoips/interface_modules/gridlines/default.yaml
modified: /geoips/schema/boundaries.yaml
modified: /geoips/schema/gridlines.yaml

Refactoring Updates#

Convert old “boundaries” YAML files to new “boundaries” interface json/yaml schema.#

From issue GEOIPS#117: 2023-04-02, Convert yaml files for boundaries interface

  • Details about change related to improvements or enhancements to the code

created: /geoips/interface_modules/boundaries/
created: /geoips/interface_modules/boundaries/default.yaml
created: /geoips/interface_modules/boundaries/tc_pmw.yaml
created: /geoips/interface_modules/boundaries/tc_visir.yaml
created: /geoips/interface_modules/boundaries/tc_windspeed.yaml

Convert old “gridlines” YAML files to new “gridlines” json/yaml schema.#

From issue NRLMMD-GEOIPS/geoips#118: 2023-04-04, convert yaml files for gridlines interface

created: /geoips/interface_modules/gridlines/default.yaml
created: /geoips/interface_modules/gridlines/tc_0p25degree.yaml
created: /geoips/interface_modules/gridlines/tc_pmw.yaml
created: /geoips/interface_modules/gridlines/tc_visir_3200km.yaml
created: /geoips/interface_modules/gridlines/tc_visir.yaml
created: /geoips/interface_modules/gridlines/tc_windspeed.yaml

Convert old “sectors_dynamic” YAML files to new “sectors_dynamic” json/yaml schema.#

From issue NRLMMD-GEOIPS/geoips#114: 2023-04-07, Add `sectors_dynamic` plugins

new file: geoips/interface_modules/sectors_dynamic/tc_1024x1024/tc_1km_1024x1024.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_1024x1024/tc_2km_1024x1024.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_1400x1400/tc_1km_1400x1400.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_1400x1400/tc_2km_1400x1400.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_1600x1600/tc_2km_1600x1600.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_1600x1600/tc_4km_1600x1600.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_256x256/tc_4km_256x256.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_512x512/tc_2km_512x512.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_512x512/tc_4km_512x512.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_800x800/tc_2km_800x800.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_800x800/tc_4km_800x800.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_huge/tc_0p1km_3200x3200.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_huge/tc_1km_2500x2500.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_huge/tc_1km_3200x3200.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_web_2km_template.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_web_ascat_high_barbs_template.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_web_ascatuhr_barbs_template.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_web_halfkm_template.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_web_qkm_template.yaml
new file: geoips/interface_modules/sectors_dynamic/tc_web_template.yaml

Convert old “product_params” YAML files to new “product_templates” json/yaml schema.#

From issue NRLMMD-GEOIPS/geoips#122: 2023-04-04, Add `product_templates` plugins

Convert all YAML files under geoips/yaml_configs/product_params to valid YAML-based plugins.

Note the interface is “product_templates” for these plugins, and the family can vary. The family determines the contents/layout of the “spec”, and the plugin format must conform with the schema defined in: geoips/schema/product_templates/<family>.yaml

new: geoips/interface_modules/product_templates/alg.yaml
new: geoips/interface_modules/product_templates/Uncorrected-Channel.yaml
new: geoips/interface_modules/product_templates/interp.yaml
new: geoips/interface_modules/product_templates/sectored.yaml
new: geoips/interface_modules/product_templates/unmodified.yaml
new: geoips/interface_modules/product_templates/unsectored.yaml
new: geoips/interface_modules/product_templates/pmw_150/150H.yaml
new: geoips/interface_modules/product_templates/pmw_150/150VNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/150V.yaml
new: geoips/interface_modules/product_templates/pmw_150/157VNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/157V.yaml
new: geoips/interface_modules/product_templates/pmw_150/165HNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/165H.yaml
new: geoips/interface_modules/product_templates/pmw_150/166HNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/166H.yaml
new: geoips/interface_modules/product_templates/pmw_150/166VNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/166V.yaml
new: geoips/interface_modules/product_templates/pmw_150/183-1HNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/183-1H.yaml
new: geoips/interface_modules/product_templates/pmw_150/183-3HNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/183-3H.yaml
new: geoips/interface_modules/product_templates/pmw_150/183-7H.yaml
new: geoips/interface_modules/product_templates/pmw_150/183HNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/183H.yaml
new: geoips/interface_modules/product_templates/pmw_150/190VNearest.yaml
new: geoips/interface_modules/product_templates/pmw_150/190V.yaml
new: geoips/interface_modules/product_templates/pmw_37/19HNearest.yaml
new: geoips/interface_modules/product_templates/pmw_37/19H.yaml
new: geoips/interface_modules/product_templates/pmw_37/19VNearest.yaml
new: geoips/interface_modules/product_templates/pmw_37/19V.yaml
new: geoips/interface_modules/product_templates/pmw_37/37H-LegacyNearest.yaml
new: geoips/interface_modules/product_templates/pmw_37/37H-Legacy.yaml
new: geoips/interface_modules/product_templates/pmw_37/37HNearest.yaml
new: geoips/interface_modules/product_templates/pmw_37/37H-PhysicalNearest.yaml
new: geoips/interface_modules/product_templates/pmw_37/37H-Physical.yaml
new: geoips/interface_modules/product_templates/pmw_37/37H.yaml
new: geoips/interface_modules/product_templates/pmw_37/37pctNearest.yaml
new: geoips/interface_modules/product_templates/pmw_37/37pct.yaml
new: geoips/interface_modules/product_templates/pmw_37/37VNearest.yaml
new: geoips/interface_modules/product_templates/pmw_37/37V.yaml
new: geoips/interface_modules/product_templates/pmw_37/color37Nearest.yaml
new: geoips/interface_modules/product_templates/pmw_37/color37.yaml
new: geoips/interface_modules/product_templates/pmw_89/89H-LegacyNearest.yaml
new: geoips/interface_modules/product_templates/pmw_89/89H-Legacy.yaml
new: geoips/interface_modules/product_templates/pmw_89/89HNearest.yaml
new: geoips/interface_modules/product_templates/pmw_89/89H-PhysicalNearest.yaml
new: geoips/interface_modules/product_templates/pmw_89/89H-Physical.yaml
new: geoips/interface_modules/product_templates/pmw_89/89HWNearest.yaml
new: geoips/interface_modules/product_templates/pmw_89/89HW.yaml
new: geoips/interface_modules/product_templates/pmw_89/89H.yaml
new: geoips/interface_modules/product_templates/pmw_89/89pctNearest.yaml
new: geoips/interface_modules/product_templates/pmw_89/89pct.yaml
new: geoips/interface_modules/product_templates/pmw_89/89VNearest.yaml
new: geoips/interface_modules/product_templates/pmw_89/89V.yaml
new: geoips/interface_modules/product_templates/pmw_89/color89Nearest.yaml
new: geoips/interface_modules/product_templates/pmw_89/color89.yaml
new: geoips/interface_modules/product_templates/rain_rate/RainNearest.yaml
new: geoips/interface_modules/product_templates/rain_rate/Rain.yaml
new: geoips/interface_modules/product_templates/sfc_winds/incident-angle.yaml
new: geoips/interface_modules/product_templates/sfc_winds/nrcs.yaml
new: geoips/interface_modules/product_templates/sfc_winds/wind-ambiguities.yaml
new: geoips/interface_modules/product_templates/sfc_winds/windbarbs.yaml
new: geoips/interface_modules/product_templates/sfc_winds/windspeed.yaml
new: geoips/interface_modules/product_templates/tpw/TPW-CIMSS.yaml
new: geoips/interface_modules/product_templates/tpw/TPW-Purple.yaml
new: geoips/interface_modules/product_templates/tpw/TPW-PWAT.yaml
new: geoips/interface_modules/product_templates/visir/Infrared-Gray.yaml
new: geoips/interface_modules/product_templates/visir/Infrared.yaml
new: geoips/interface_modules/product_templates/visir/IR-BD.yaml
new: geoips/interface_modules/product_templates/visir/Night-Vis-GeoIPS1.yaml
new: geoips/interface_modules/product_templates/visir/Night-Vis-IR-GeoIPS1.yaml
new: geoips/interface_modules/product_templates/visir/Night-Vis-IR.yaml
new: geoips/interface_modules/product_templates/visir/Night-Vis.yaml
new: geoips/interface_modules/product_templates/visir/Visible.yaml
new: geoips/interface_modules/product_templates/visir/WV-Lower.yaml
new: geoips/interface_modules/product_templates/visir/WV-Upper.yaml
new: geoips/interface_modules/product_templates/visir/WV.yaml

Testing Updates#

Add unit tests to validate yaml-based plugins and schema#

From issue NRLMMD-GEOIPS/geoips#151: 2023-04-06, Add unit tests to validate plugins

  • Added pytest requirements to pyproject.toml

  • Moved bad plugin examples to tests/pytests

  • Added test script to validate all good YAML plugins, as well as confirm bad yaml plugins fail.

  • Add pytest tests/pytests to test_full_install.sh

modified: tests/test_full_install.sh
created: tests/test_pytest/test_yaml_plugins.py
modified: pyproject.toml
moved: bad_schema -> tests/pytests/bad_plugins