| # # # Distribution Statement A. Approved for public release. Distribution unlimited. | # # # | # # # Author: | # # # Naval Research Laboratory, Marine Meteorology Division | # # # | # # # This program is free software: you can redistribute it and/or modify it under | # # # the terms of the NRLMMD License included with this program. This program is | # # # distributed WITHOUT ANY WARRANTY; without even the implied warranty of | # # # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the included license | # # # for more details. If you did not receive the license, for more information see: | # # # https://github.com/U-S-NRL-Marine-Meteorology-Division/ Version 1.10.0a0 (2023-04-04) ***************************** First attempt at YAML-based plugin interfaces. These updates will be superseded by later alpha releases of v1.10.0. Tagging this version for reference. * Improve output from estimate area extent script * Add YAML-based plugin validator * Update plugin validator to follow references and add a base schema * Update plugin validator to use interface.family or interface * Add new plugin schema * "boundaries" plugin schema * "gridlines" plugin schema * "product_templates" plugin schema (for each of 11 families) * "sectors_dynamic" plugin schema * Update boundaries and gridlines schema to better handle defaults * Convert old YAML to new formats * boundaries YAML to new format * gridlines YAML to new format * product_params YAML to new product_templates format * sectors_dynamic YAML to new format * Add unit tests to validate yaml-based plugins and schema Major New Functionality ======================= Add a validator for YAML-based plugins. --------------------------------------- *From issue NRLMMD-GEOIPS/geoips#132: 2023-04-04, Add validator for YAML plugins.* :: created: /geoips/schema/__init__.py created: /geoips/schema/plugin_validator.py Add a schema for validating `Gridlines` plugins. ------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#109: 2023-04-04, Add schema for Gridlines plugins.* :: modified: /geoips/schema/gridlines.yaml Add a schema for validating `Boundaries` plugins. ------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#108: 2023-04-04, Add schema for `Boundaries` plugins.* :: modified: /geoips/schema/boundaries.yaml Add schema for validating `sectors_dynamic` plugins --------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#114: 2023-04-07, Add `sectors_dynamic` plugins* :: new file: geoips/schema/sectors_dynamic.yaml Add schema for validating `product_templates` plugins ----------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#122: 2023-04-04, Add `product_templates` plugins* product_templates plugins require family-specific schema - each family within the interface has a unique schema for validation. These schema are added to the "product_templates" subdirectory under geoips/schema :: new: geoips/interface_modules/product_templates/alg_cmap.yaml new: geoips/interface_modules/product_templates/alg_interp_cmap.yaml new: geoips/interface_modules/product_templates/alg.yaml new: geoips/interface_modules/product_templates/interp_alg_cmap.yaml new: geoips/interface_modules/product_templates/interp_alg.yaml new: geoips/interface_modules/product_templates/interp.yaml Note the following families have "additionalProprerties: true" set on the spec, which means the actual contents of the products are not validated at all. We will likely want to update these in the future to be named "unvalidated_*" to make it clear that they are intended to be used as temporary product families during development, and once the format of the family is fully identified, we can define a new family to comply. :: new: geoips/interface_modules/product_templates/sectored_xarray_dict_to_output_format.yaml new: geoips/interface_modules/product_templates/unmodified.yaml new: geoips/interface_modules/product_templates/unsectored_xarray_dict_area_to_output_format.yaml new: geoips/interface_modules/product_templates/unsectored_xarray_dict_to_output_format.yaml new: geoips/interface_modules/product_templates/xarray_dict_to_output_format.yaml Bug Fixes ========= Don't rely on environment variables in unit tests ------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#156: 2023-04-06, Don't rely on environment variables in unit tests* * Switched from using GEOIPS_PACKAGES_DIR to importlib.resources.files :: modified: tests/test_pytest/test_yaml_plugins.py Enhancements ============ Update output from estimate area extent script. ----------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#123: 2023-04-06, update estimate_area_extent* * estimate_area_extent script now provides copy/pastable formatted yaml * Change sector_type to sector_family :: modified: geoips/sector_utils/estimate_area_extent.py Update validator to use both interface and family names ------------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#122: 2023-04-04, Add `product_templates` plugins* This parses the path to the schema file in order to determine the interface and family name. geoips/schema/.yaml for interface specific schema. geoips/schema//.yaml for family-specific schema. :: modified: /geoips/schema/plugin_validator.py Update plugin validator to follow references and add a base schema for all schema --------------------------------------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#155: 2023-04-06, Follow references in plugin validator* :: modified: /geoips/schema/plugin_validator.py modified: /geoips/schema/base.yaml modified: /geoips/schema/gridlines.yaml modified: /geoips/schema/boundaries.yaml Update boundaries and gridlines schema to simplify and handle default values better ----------------------------------------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#144: 2023-04-05, Update boundaries/gridlines schema* :: modified: /geoips/interface_modules/boundaries/default.yaml modified: /geoips/interface_modules/gridlines/default.yaml modified: /geoips/schema/boundaries.yaml modified: /geoips/schema/gridlines.yaml Refactoring Updates =================== Convert old "boundaries" YAML files to new "boundaries" interface json/yaml schema. ----------------------------------------------------------------------------------- *From issue GEOIPS#117: 2023-04-02, Convert yaml files for boundaries interface* * Details about change related to improvements or enhancements to the code :: created: /geoips/interface_modules/boundaries/ created: /geoips/interface_modules/boundaries/default.yaml created: /geoips/interface_modules/boundaries/tc_pmw.yaml created: /geoips/interface_modules/boundaries/tc_visir.yaml created: /geoips/interface_modules/boundaries/tc_windspeed.yaml Convert old "gridlines" YAML files to new "gridlines" json/yaml schema. ----------------------------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#118: 2023-04-04, convert yaml files for gridlines interface* :: created: /geoips/interface_modules/gridlines/default.yaml created: /geoips/interface_modules/gridlines/tc_0p25degree.yaml created: /geoips/interface_modules/gridlines/tc_pmw.yaml created: /geoips/interface_modules/gridlines/tc_visir_3200km.yaml created: /geoips/interface_modules/gridlines/tc_visir.yaml created: /geoips/interface_modules/gridlines/tc_windspeed.yaml Convert old "sectors_dynamic" YAML files to new "sectors_dynamic" json/yaml schema. ----------------------------------------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#114: 2023-04-07, Add `sectors_dynamic` plugins* :: new file: geoips/interface_modules/sectors_dynamic/tc_1024x1024/tc_1km_1024x1024.yaml new file: geoips/interface_modules/sectors_dynamic/tc_1024x1024/tc_2km_1024x1024.yaml new file: geoips/interface_modules/sectors_dynamic/tc_1400x1400/tc_1km_1400x1400.yaml new file: geoips/interface_modules/sectors_dynamic/tc_1400x1400/tc_2km_1400x1400.yaml new file: geoips/interface_modules/sectors_dynamic/tc_1600x1600/tc_2km_1600x1600.yaml new file: geoips/interface_modules/sectors_dynamic/tc_1600x1600/tc_4km_1600x1600.yaml new file: geoips/interface_modules/sectors_dynamic/tc_256x256/tc_4km_256x256.yaml new file: geoips/interface_modules/sectors_dynamic/tc_512x512/tc_2km_512x512.yaml new file: geoips/interface_modules/sectors_dynamic/tc_512x512/tc_4km_512x512.yaml new file: geoips/interface_modules/sectors_dynamic/tc_800x800/tc_2km_800x800.yaml new file: geoips/interface_modules/sectors_dynamic/tc_800x800/tc_4km_800x800.yaml new file: geoips/interface_modules/sectors_dynamic/tc_huge/tc_0p1km_3200x3200.yaml new file: geoips/interface_modules/sectors_dynamic/tc_huge/tc_1km_2500x2500.yaml new file: geoips/interface_modules/sectors_dynamic/tc_huge/tc_1km_3200x3200.yaml new file: geoips/interface_modules/sectors_dynamic/tc_web_2km_template.yaml new file: geoips/interface_modules/sectors_dynamic/tc_web_ascat_high_barbs_template.yaml new file: geoips/interface_modules/sectors_dynamic/tc_web_ascatuhr_barbs_template.yaml new file: geoips/interface_modules/sectors_dynamic/tc_web_halfkm_template.yaml new file: geoips/interface_modules/sectors_dynamic/tc_web_qkm_template.yaml new file: geoips/interface_modules/sectors_dynamic/tc_web_template.yaml Convert old "product_params" YAML files to new "product_templates" json/yaml schema. ------------------------------------------------------------------------------------ *From issue NRLMMD-GEOIPS/geoips#122: 2023-04-04, Add `product_templates` plugins* Convert all YAML files under geoips/yaml_configs/product_params to valid YAML-based plugins. Note the interface is "product_templates" for these plugins, and the family can vary. The family determines the contents/layout of the "spec", and the plugin format must conform with the schema defined in: geoips/schema/product_templates/.yaml :: new: geoips/interface_modules/product_templates/alg.yaml new: geoips/interface_modules/product_templates/Uncorrected-Channel.yaml new: geoips/interface_modules/product_templates/interp.yaml new: geoips/interface_modules/product_templates/sectored.yaml new: geoips/interface_modules/product_templates/unmodified.yaml new: geoips/interface_modules/product_templates/unsectored.yaml new: geoips/interface_modules/product_templates/pmw_150/150H.yaml new: geoips/interface_modules/product_templates/pmw_150/150VNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/150V.yaml new: geoips/interface_modules/product_templates/pmw_150/157VNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/157V.yaml new: geoips/interface_modules/product_templates/pmw_150/165HNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/165H.yaml new: geoips/interface_modules/product_templates/pmw_150/166HNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/166H.yaml new: geoips/interface_modules/product_templates/pmw_150/166VNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/166V.yaml new: geoips/interface_modules/product_templates/pmw_150/183-1HNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/183-1H.yaml new: geoips/interface_modules/product_templates/pmw_150/183-3HNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/183-3H.yaml new: geoips/interface_modules/product_templates/pmw_150/183-7H.yaml new: geoips/interface_modules/product_templates/pmw_150/183HNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/183H.yaml new: geoips/interface_modules/product_templates/pmw_150/190VNearest.yaml new: geoips/interface_modules/product_templates/pmw_150/190V.yaml new: geoips/interface_modules/product_templates/pmw_37/19HNearest.yaml new: geoips/interface_modules/product_templates/pmw_37/19H.yaml new: geoips/interface_modules/product_templates/pmw_37/19VNearest.yaml new: geoips/interface_modules/product_templates/pmw_37/19V.yaml new: geoips/interface_modules/product_templates/pmw_37/37H-LegacyNearest.yaml new: geoips/interface_modules/product_templates/pmw_37/37H-Legacy.yaml new: geoips/interface_modules/product_templates/pmw_37/37HNearest.yaml new: geoips/interface_modules/product_templates/pmw_37/37H-PhysicalNearest.yaml new: geoips/interface_modules/product_templates/pmw_37/37H-Physical.yaml new: geoips/interface_modules/product_templates/pmw_37/37H.yaml new: geoips/interface_modules/product_templates/pmw_37/37pctNearest.yaml new: geoips/interface_modules/product_templates/pmw_37/37pct.yaml new: geoips/interface_modules/product_templates/pmw_37/37VNearest.yaml new: geoips/interface_modules/product_templates/pmw_37/37V.yaml new: geoips/interface_modules/product_templates/pmw_37/color37Nearest.yaml new: geoips/interface_modules/product_templates/pmw_37/color37.yaml new: geoips/interface_modules/product_templates/pmw_89/89H-LegacyNearest.yaml new: geoips/interface_modules/product_templates/pmw_89/89H-Legacy.yaml new: geoips/interface_modules/product_templates/pmw_89/89HNearest.yaml new: geoips/interface_modules/product_templates/pmw_89/89H-PhysicalNearest.yaml new: geoips/interface_modules/product_templates/pmw_89/89H-Physical.yaml new: geoips/interface_modules/product_templates/pmw_89/89HWNearest.yaml new: geoips/interface_modules/product_templates/pmw_89/89HW.yaml new: geoips/interface_modules/product_templates/pmw_89/89H.yaml new: geoips/interface_modules/product_templates/pmw_89/89pctNearest.yaml new: geoips/interface_modules/product_templates/pmw_89/89pct.yaml new: geoips/interface_modules/product_templates/pmw_89/89VNearest.yaml new: geoips/interface_modules/product_templates/pmw_89/89V.yaml new: geoips/interface_modules/product_templates/pmw_89/color89Nearest.yaml new: geoips/interface_modules/product_templates/pmw_89/color89.yaml new: geoips/interface_modules/product_templates/rain_rate/RainNearest.yaml new: geoips/interface_modules/product_templates/rain_rate/Rain.yaml new: geoips/interface_modules/product_templates/sfc_winds/incident-angle.yaml new: geoips/interface_modules/product_templates/sfc_winds/nrcs.yaml new: geoips/interface_modules/product_templates/sfc_winds/wind-ambiguities.yaml new: geoips/interface_modules/product_templates/sfc_winds/windbarbs.yaml new: geoips/interface_modules/product_templates/sfc_winds/windspeed.yaml new: geoips/interface_modules/product_templates/tpw/TPW-CIMSS.yaml new: geoips/interface_modules/product_templates/tpw/TPW-Purple.yaml new: geoips/interface_modules/product_templates/tpw/TPW-PWAT.yaml new: geoips/interface_modules/product_templates/visir/Infrared-Gray.yaml new: geoips/interface_modules/product_templates/visir/Infrared.yaml new: geoips/interface_modules/product_templates/visir/IR-BD.yaml new: geoips/interface_modules/product_templates/visir/Night-Vis-GeoIPS1.yaml new: geoips/interface_modules/product_templates/visir/Night-Vis-IR-GeoIPS1.yaml new: geoips/interface_modules/product_templates/visir/Night-Vis-IR.yaml new: geoips/interface_modules/product_templates/visir/Night-Vis.yaml new: geoips/interface_modules/product_templates/visir/Visible.yaml new: geoips/interface_modules/product_templates/visir/WV-Lower.yaml new: geoips/interface_modules/product_templates/visir/WV-Upper.yaml new: geoips/interface_modules/product_templates/visir/WV.yaml Testing Updates =============== Add unit tests to validate yaml-based plugins and schema -------------------------------------------------------- *From issue NRLMMD-GEOIPS/geoips#151: 2023-04-06, Add unit tests to validate plugins* * Added pytest requirements to pyproject.toml * Moved bad plugin examples to tests/pytests * Added test script to validate all good YAML plugins, as well as confirm bad yaml plugins fail. * Add pytest tests/pytests to test_full_install.sh :: modified: tests/test_full_install.sh created: tests/test_pytest/test_yaml_plugins.py modified: pyproject.toml moved: bad_schema -> tests/pytests/bad_plugins