Version 1.12.0 (2023-12-07)#

Allow passing plugin_package name to create_plugin_registries#

From GEOIPS#422: 2023-12-14, 1.12.0 release bug fixes

Either create all plugin registries (one argument), or a single registry (pass package name to create_plugin_registries)

modified: geoips/create_plugin_registries.py

Call identify_checker and get_plugin separately for output_checkers#

From GEOIPS#422: 2023-12-14, 1.12.0 release bug fixes

• Remove EntryPointError

  • EntryPointError no longer needed (not using EntryPoints for individual plugins)

  • Removed from interfaces/base.py - get_plugin should never raise EntryPointError

• Remove find_entry_point, list_entry_points, and get_all_entry_points

  • No longer used since entry points are not used for individual plugins.

  • Plugin registries are now used exclusively for finding plugins, so we do not need individual entry points (only a single entry point for each plugin package).

• Call identify checker and get_plugin separately from procflows

  • Rather than having a special case in base.py get_plugin and output_checkers get_plugin which allowed passing in full paths to output products and automatically identifying the checker name and opening it, just explicitly call identify_checker and get_plugin separately from the procflows.

  • single_source, config_based, and data_fusion procflows all now call identify_checker on the output product filename, and get_plugin on the resulting checker plugin name.

  • This simplified a lot of logic in both output_checkers and interfaces/base.py. Supporting passing a filename to output_checkers.get_plugin() was unnecessarily complicated.

• Added log.interactive when sector has no coverage

M geoips/errors.py
M geoips/geoips_utils.py
M geoips/interfaces/base.py
M geoips/interfaces/module_based/output_checkers.py
M geoips/plugins/modules/procflows/single_source.py
M geoips/plugins/modules/procflows/config_based.py

Allow filtering products based on platform name#

From GEOIPS#343: 2023-12-02, filter products by platform

Allow specifying a list of products to either include or exclude on a per-product basis.

In plot_data, if the alg_xarray platform name IS in exclude_platforms, or IS NOT in include_platforms if it exists, then DO NOT plot the current product.

modified: geoips/plugins/modules/procflows/single_source.py

If plot_data does not return any products, then do not append to the final products list and continue.

modified: geoips/plugins/modules/procflows/config_based.py
modified: geoips/plugins/modules/procflows/single_source.py

Updated SSMI/S product YAML to exclude 150H (ch8). This channel has been bad since shortly after launch.

modified: geoips/plugins/yaml/products/ssmis.yaml

Added some more informative output to bdeck parser for incorrectly formatted files in the process:

modified: geoips/plugins/modules/sector_metadata_generators/bdeck_parser.py

Add exclude_platforms and include_platforms options to low memory abi procflow config. Currently only the exclude_platforms option is tested through the new test script, but include both for reference.

modified: tests/yaml_configs/abi_test_low_memory.yaml
new: tests/scripts/abi.config_based_exclude_goes17.sh

BUG FIX: added a missing found_one variable to missing products check.

modified: geoips/interfaces/module_based/output_checkers.py

Rename JPSS-01/02 platform name to noaa-20/21 in ATMS reader#

From GEOIPS#394: 2023-11-08, atms platform names

JPSS-01 and JPSS-02 were renamed to noaa-20 and noaa-21.

• Updating platform name in ATMS reader will make naming consistent with VIIRS reader.

modified: geoips/plugins/modules/readers/atms_hdf5.py

Add hy-2d platform to scat knmi winds reader#

From GEOIPS#418: 2023-11-30, add hy2

Added HY-2D HSCAT platform to supported platforms in scat_knmi_winds_netcdf reader. Maps internal attributes to GeoIPS platform and source name.

geoips/plugins/modules/readers/scat_knmi_winds_netcdf.py

Fix build_docs.sh geoips template paths#

Had accidentally broken these when trying to get actions to work

docs/build_docs.sh

Check for unit_tests directory before running pytest in check_code.sh#

This was failing on repos with no unit tests directories.

modified: tests/utils/check_code.sh

Update test_dtree to compare attributes rather than full datatree#

Due to xarray datatree package update, must now compare attributes on datatree rather than full datatree. pytests began failing due to updated datatree package.

modified: tests/unit_tests/xarray_utils/test_dtree.py

Ensure create_plugin_registries fails on bad plugins#

From GEOIPS#415: 2023-11-27, registry fail on bad plugins

Updated create_plugin_registries to reliably error on improperly formatted plugins. ALL python plugins now MUST include the interface variable at the top level, and if interface != None, then interface, family, and name must ALL be specified. This allows us to always fail on an explicit error in a python module within the plugins directory, so we can ensure no files are silently skipped. Previously any plugin that raised an ImportError or did not contain interface, family, and name would just be silently skipped (since we were assuming they were not intended to be a valid plugin), but this would result in any plugin that happened to have a typo in interface, family, name, or had a legitimate ImportError at the top level to be silently skipped with no feedback or indication that there may be a problem. Now we are very explicit about the error catching, and ONLY skip files that explicitly have interface = None set at the top level. Any other errors will be caught, raised, and reported.

Note this may not be the final layout for handling non-plugins, discussion: NRLMMD-GEOIPS/geoips#407

Added a number of comments (largely regarding the new interface = None requirement, as well as clarification on appending string error messages to the error_message variable to raise once at the end of processing)

modified: geoips/create_plugin_registries.py

Update geoips/interfaces/base.py and geoips/create_plugin_registries.py to allow relative imports within modules. Previously the spec.loader.exec_module function was passed just the module_name, which caused relative imports within modules to fail, because exec_module could not resolve the full path since no parent was specified. This resolves the ImportError for the seviri reader.

modified: geoips/interfaces/base.py
modified: geoips/create_plugin_registries.py

Remove all individual plugin entry points from pyproject.toml (only require a single plugin_package entry point now, since all plugins are discovered during the create_plugin_registries phase. Console scripts will also still need to be defined if required).

modified: pyproject.toml

Add interface = None to all utility helper modules in the plugins directory.

modified: geoips/plugins/modules/filename_formatters/utils/tc_file_naming.py
modified: geoips/plugins/modules/interpolators/utils/boxdefinitions.py
modified: geoips/plugins/modules/interpolators/utils/interp_pyresample.py
modified: geoips/plugins/modules/interpolators/utils/interp_scipy.py
modified: geoips/plugins/modules/readers/utils/geostationary_geolocation.py
modified: geoips/plugins/modules/readers/utils/hrit_reader.py
modified: geoips/plugins/modules/readers/utils/remss_reader.py

Fix GEOIPS_VERS in config_geoips setup file#

Sort tags by creation date to identify latest version

modified: setup/config_geoips

Add extensive YAML procflow config, including many options#

new file: scripts/amsr2.config_based_no_compare_full.sh
new file: yaml_configs/amsr2_no_compare_full.yaml

Use explicit list of valid interfaces for validating plugin_registries#

Explicitly set module_based_interfaces and yaml_based_interfaces lists in geoips/interfaces/__init__.py (and combine them to set __all__), then use those lists to determine the valid plugin interfaces during plugin registry validation in geoips/plugin_registry.py. The same format can be used from other repositories to enable plugin interface name validation for interfaces defined in other plugin packages. Additionally, ignore flake8 error F401 in */interfaces/__init__.py, unused import, since the interface names are no longer set directly in __all__ they are not recogzied as used.

modified: geoips/interfaces/__init__.py
modified: geoips/plugin_registry.py
modified: .config/flake8

Separate long and short running unit tests in check_code.sh#

Previously ran ALL tests in geoips/tests, now have pytest_short and pytest_long options to run geoips/tests/unit_tests and geoips/tests/unit_tests_long. pytest is still called directly from the workflows, but this ensures check_code.sh works most closely to how the GitHub Actions workflows operate (since pytest-short workflow only operates on unit_tests directory, ensure you can break up pytests from check_code.sh as well.)

modified: tests/utils/check_code.sh

Remove long running unit tests from output checkers#

Currently all unit tests under output_checkers interface are run with the “short” running unit tests. The developer should be able to specify individual unit tests under a given interface to allow “long” running unit tests (so they are not run with the short running tests).

Currently, the standard unit testing function names have been modified to include “_long” - and within the output checkers unit testing infrastructure any unit testing functions named get_test_files_long and perform_test_comparison_long are explicitly xfailed, with the intent to implement long running output comparisons in the future.

Additional discussion / review required before fully implementing the long/short running unit tests for each interface, so currently just disabling these tests.

Note also the real reason for disabling the geotiff tests is that they are using test data found in the “tests/outputs” directory, which is not available when pip installing geoips from GitHub Actions. This is another discussion point for review, as we could either continue running the geotiff output checker as written, and just assume tests/outputs will be available for the “long” running unit tests. Or we could disallow using tests/outputs within unit tests. But for now, this unit test will just be disabled by xfailing on “long” unit tests.

modified: geoips/plugins/modules/output_checkers/geotiff.py
modified: tests/unit_tests/plugins/modules/output_checkers/test_output_checkers.py

Allow gunzipping current output to scratch location during tests#

From GEOIPS#412: 2023-11-27, gunzip in output checkers

Add clobber gunzip to get_compare_products OutputCheckersBasePlugin method (first time attempting to access the comparison product), and clobber gunzip to OutputCheckersInterface identify_checker method (ie, first time attempting to access the current output product). All subsequent gunzip_product calls use clobber=False (so we only gunzip once per run, but we always gunzip the first time through).

Create individual functions for each piece of the output comparison checks and reporting. Call these individual functions and methods from compare_outputs method (all of this stuff used to be directly in compare_outputs).

• Create the files that are sourced for copying/deleting files to update test repos

  • write_bad_comparisons_to_file (cp_BADCOMPARES.txt) - update test outputs that were modified, but not renamed

  • write_remove_temp_files_to_file (rm_TEMPFILES.txt) - remove temporarily uncompressed files from scratch location

  • write_missing_products_to_file (rm_MISSINGPRODUCTS.txt) - remove files from test output location that did NOT existing in the output from the current run

  • write_missing_comparisons_to_file (cp_MISSINGCOMPARE.txt) - copy files from output path of current run to test output directory - files that did NOT exist in the test output path, but were found in the current output.

  • write_good_comparisons_to_file (cp_GOODCOMPARE.txt) - copy files from output of current run that matched test output files exactly to a single directory. This is just for convenience (so all outputs are in one place), and not helpful for keeping test outputs up to date.

• Consolidate log_with_emphasis function for outputting asterisks before and after list of log output lines.

• Functions for identifying missing and bad output files

  • get_missing_products function, compare list of current outputs to dictionary of comparison test outputs - return list of products that were found in the current run but NOT found in the comparison test output directory.

    • NOTE: this must handle gunzipped test outputs correctly!

  • get_compare_products method - get a dictionary containing all the comparison products in the comparison test output path. This handles gunzipping the comparison products as required, and storing the full paths to the original stored_comparison file, the gunzipped_comparison file, and the file_for_comparison that should actually be used for the image comparisons. Returns the dictionary of comparison products, and a list of temp files that will need to be removed (written to scratch directory).

  • perform_comparisons method - compare the list of output_products from the current run to the dictionary of compare_products in the test outputs directory. Return all associated good, bad, missing comparison file lists.

• Functions for gunzipping - these were previously methods, move to functions so generally accessible.

  • is_gz function - Call from is_gz method.

  • gunzip_product function - gunzip to $GEOIPS_OUTDIRS scratch directory, separate sub directories for comparison and output products. “clobber” argument for forcing overwrite of existing file (typically do NOT overwrite). Call from gunzip_product method.

  • Remove gzip_product method - no longer required.

Note many of these functions and methods use a diffdir argument, which is used for both writing out the diff images and scripts, as well as determining the final path for the updated output comparison files (relative to diffdir). This will later be updated to separate the diffdir from the test output comparison path, allowing us to specify the temporary output directory for the diff images and scripts separately from the actual test output comparison path.

• temp_path - will be diff images and scripts

• compare_path - will be path to the test output comparison files

modified: geoips/interfaces/module_based/output_checkers.py

Update image list to just png, jpg, and jpeg extensions.

modified: geoips/plugins/modules/output_checkers/image.py

Remove all references to gunzipping test outputs in source repos from check_system_requirements.sh. NOTE gunzipping will still happen in test data repos, but not for test outputs (anything that is gzipped in the test output datasets will automatically be gunzipped during the testing process. test data repositories are still gunzipped in advance as needed).

Also remove the uncompress_test_data.sh script from tests/outputs - no longer required.

removed: tests/uncompress_test_data.sh
modified: setup/check_system_requirements.sh

Update compare_outputs to print netcdf attr diffs to logs#

From GEOIPS#394: 2023-11-08, atms platform names

Ensure netcdf attribute diffs are printed at the interactive level, was still printing to info level. Also, move dividing asterisk line to after the attr diffs.

geoips/compare_outputs.py

Rename JPSS-01 platform name to noaa-20 in ATMS test output#

From GEOIPS#394: 2023-11-08, atms platform names From GEOIPS#394: 2023-11-30, atms platform names

Replaced “J01” with “noaa-20” in ATMS netCDF test output file name. I think this was renamed 8 Nov, then attrs updated 30 Nov.

renamed:    tests/outputs/atms.tc.165H.netcdf_geoips/20210809.083826.J01.165H_latitude_longitude.tc2021ep11kevin.nc -> tests/outputs/atms.tc.165H.netcdf_geoips/20210809.083826.noaa-20.165H_latitude_longitude.tc2021ep11kevin.nc
modified: tests/outputs/atms.tc.165H.netcdf_geoips/20210809.083826.noaa-20.165H_latitude_longitude.tc2021ep11kevin.nc

Add test_interfaces GitHub Actions Workflow#

From GEOIPS#342: 2023-12-01, github actions test interfaces

Sets up python enviroment and pip installs geoips. Then runs the check_code.sh interfaces command. Note this command operates on all installed plugins, so only needs to be called once from geoips_dev_utils, and not separately from each plugin repo.

Activate python environment for each step.

.github/workflows/test-interfaces.yaml

Add pytest GitHub Actions Workflow#

From GEOIPS#342: 2023-12-01, github actions test interfaces

Sets up python enviroment and pip installs geoips. Then runs the “short” pytest unit tests in tests/unit_tests in both the geoips repo and the current plugin repo. If the current plugin repo is geoips, only run pytest once.

Activate python environment for each step.

.github/workflows/pytest.yaml

Add build_docs GitHub Actions Workflow#

From GEOIPS#342: 2023-12-01, github actions test interfaces

Sets up python enviroment and pip installs geoips. Initial attempt at running docs/build_docs.sh - not yet fully functional, will finalize in a separate PR.

First step in updating build_docs.sh required removing the GEOIPS_PACKAGES_DIR environment variable (which also involved removing GEOIPS_PACKAGES_DIR from any required environment variables in base_paths.py)

new: .github/workflows/build_docs_html.yaml
modified: docs/build_docs.sh
modified: geoips/filenames/base_paths.py

Add release note for v1.12.0#

From GEOIPS#376: 2023-11-01, release process updates

From GEOIPS#408: 2023-11-19, release process updates

All updates until the next release (v1.12.0) will be included in this release note.

modified: docs/source/releases/v1_12_0.rst
modified: docs/source/releases/index.rst

Update ‘update_this_release_note’#

From GEOIPS#376: 2023-11-01, release process updates

From GEOIPS#408: 2023-11-19, release process updates

All updates until the next release (v1.12.0) will be included in the release note specified in this file.

modified: update_this_release_note

Update imports related to geoips database modules#

All geoips_db plugins now accessed through class-based geoips_db interface.

modified: geoips/plugins/modules/procflows/single_source.py