Version 1.18.1 (2026-07-23)#
Release process: Add 1.18.0 release note
Release process: Update internal and github.com ‘upcoming_version’ and ‘tagged_version’
Bug fixes: Ensure we error appropriately if any steps in the base, full, or site install fail.
Bug fixes: Update image output checker so the diff image mode matches the input image modes
Bug fixes: More informative error output for improperly formatted plugins
Bug fixes: Add missing open source packages to full and site install.
Bug fixes: Allow spec to be defined in product_defaults (not product) in pydantic model
Bug fixes: Remove validation of text file format during output checking
Bug fixes: Updating dataframe resource usage parsing to check if the sector value is a string before taking a substring
Bug fixes: Update base_paths.py so all global env vars are cast to bool/none appropriately
Bug fixes: Add test_data_sgli to full_install.sh
Bug fixes: Add http.version http/1.1 to gitconfig
Bug fixes: Add additional supported text output checkers formats
Continuous integration: Update code base for black 2026 style - see black 26.1.0 release notes for details.
Enhancement: Add doc8 linting to code testing framework
Enhancement: Class based plugins – registry updates
Enhancement: Rewrite of text output checker
Enhancement: Consolidate yaml-based and module-based unit tests into test_pydanitc_models.py
Enhancement: Adding an output checker pydantic model
Enhancement: Add no_color environment variable to geoips
Enhancement: Order based – dealing with generated and non-registered workflow plugins
Enhancement: Base plugin classes
Enhancement: Obp pydantic product / product default models
Enhancement: Add output to geoips config delete-registries
Realtime: Update config_based procflow to pull sector_adjuster_kwargs from output config yaml
Testing: Test data abi dataset update v1.17.1
Testing: Cli unit tests – test dataset name updates
Testing: Allow importing validation and lint test lists from geoips repo, and add validation pytest markers
Testing: Creating tiny sector for testing
Testing: Have fail on missing data default to true
Testing: Allow disabling dumping all failed log contents to terminal
Testing: Consolidated env variable naming for all geoips_test env vars
Testing: Temporarily disable doc8 by default for check_code.sh
Testing: Remove print statements in unit tests
Feature: Order-based procflow: add title formatters
Feature: Order-based procflow: add coverage checkers
Plotting: Histogram colorbar and static sector conformity.
Documentation: Fixed docstrings on sgli reader and on hncc algorithm.
Documentation: Add release note for version 1.18.0
Documentation: Update upcoming and tagged versions
Temporary bug fix: Temporary fix for incorrect solar and satellite azimuth angles
Minor bug fix: Add storm_id to fields returned by f-deck parser
Bug fix: Fix plugin registry rebuild check
Setup: Remove geoips_vers
Resource usage statistics: Add option to write statistics for a config_based procflow write to a json file
Resource usage statistics: Move all memusg utilities to utils/memusg
Resource usage statistics: Add json based resource usage plotting capabilities
Release process#
Add 1.18.0 release note#
Currently tagged internal version 1.17.3.
geoips#1029 - 1.18.0 updates*
added: docs/source/releases/1.18.0/1.18.0-internal-release.yaml
Update internal and github.com ‘upcoming_version’ and ‘tagged_version’#
Updated ‘update_this_release_note’ file in geoips repo for the next internal version- 1.18.1 next github.com version- 1.19.0
geoips#1029 - 1.18.0 updates*
modified: .github/versions/upcoming_version
modified: .github/versions/tagged_version
Bug fixes#
Ensure we error appropriately if any steps in the base, full, or site install fail.#
Wrap all installation commands with the check_system_requirements script (which ensures we exit non-zero if a command fails, and do not continue with the install)
Update geoips config install-github to exit non-zero if the command fails.
geoips#1029 - 1.18.0 updates
modified: tests/integration_tests/base_install.sh
modified: tests/integration_tests/full_install.sh
modified: tests/integration_tests/site_install.sh
modified: geoips/commandline/geoips_config.py
Update image output checker so the diff image mode matches the input image modes#
With 12.1.0 pillow version, RGB raw output image mode will fail when comparing against RGBA input images. Update hard coded RGB output diff image mode to use the input_image.mode to ensure they are consistent for the pixelmatch call.
geoips#1029 - 1.18.0 updates
modified: geoips/plugins/classes/output_checkers/image.py
More informative error output for improperly formatted plugins#
Ensure it is obvious which plugin is failing when validating interfaces/plugins.
geoips#1029 - 1.18.0 updates
modified: geoips/interfaces/class_based/output_checkers.py
Add missing open source packages to full and site install.#
Ensure all packages are regularly installed and tested.
test_data_abi and ahi
geoips_proxyvis
geoips_nucaps
test_data_sgli
others
geoips#1029 - 1.18.0 updates
modified: tests/integration_tests/full_install.sh
modified: tests/integration_tests/site_install.sh
Allow spec to be defined in product_defaults (not product) in pydantic model#
Also improve error messaging when undefined.
geoips#1029 - 1.18.0 updates
modified: geoips/pydantic_models/v1/products.py
Remove validation of text file format during output checking#
‘correct_file_format’ only needs to be used when attempting to auto-detect an output checker, it should not be used to validate the correct format when actually running the ‘outputs_match’ method. We are assuming when we get to the ‘outputs_match’ method that we know the files are of the correct format, and we can just compare. This allows using an explicit output checker with the –output_checker_name argument, without actually attempting to validate the format.
geoips#1029 - 1.18.0 updates
modified: geoips/plugins/classes/output_checkers/text.py
Updating dataframe resource usage parsing to check if the sector value is a string before taking a substring#
Previously only checked “if x”, now check if isinstance(x, str) to ensure it is a string rather than a float. Sometimes dataframe values can be nan, which are not caught by if x.
geoips#1029 - 1.18.0 updates
modified: geoips/utils/memusg/dataframe_utils.py
Update base_paths.py so all global env vars are cast to bool/none appropriately#
Added a new function to specifically cast explicit strings to True, False, or None, and just return the original value for all other values. Call this new function from the get_env_var call for the global variable defaults in base_paths.py.
99% sure the boolean-based env vars in the global env var defaults dict would not have actually been set to booleans, since the get_env_var function was previously not casting strings to boolean. The incorrectly specified env vars included
GEOIPS_TEST_PRINT_TEXT_OUTPUT_CHECKER_TO_CONSOLE
GEOIPS_RICH_CONSOLE_OUTPUT
GEOIPS_PROMPT_TO_OVERWRITE_COMPARISON_FILE_IF_MISMATCH
Note this means we can’t have variables defined in the geoips global defaults dictionary with the explicit strings of case insensitive “true”, “false”, “none”, or “null”, as they will be cast to True, False, None, or None respectively.
geoips#1029 - 1.18.0 updates
modified: tests/integration_tests/test_integration.py
modified: geoips/filenames/base_paths.py
Add test_data_sgli to full_install.sh#
test_data_sgli included in geoips tests/scripts.
geoips#1029 - 1.18.0 updates
modified: tests/integration_tests/full_install.sh
Add http.version http/1.1 to gitconfig#
Resolves git clone timeout issues. Note this setup is not executed if .gitconfig already exists, so will not impact existing gitconfig files.
geoips#1029 - 1.18.0 updates
modified: setup/bash_setup/gitconfigs
Add additional supported text output checkers formats#
Add .csv, .geojson, and .json output format extensions for the text output checker.
geoips#1029 - 1.18.0 updates
modified: geoips/plugins/classes/output_checkers/text.py
Continuous integration#
Update code base for black 2026 style - see black 26.1.0 release notes for details.#
Please make sure you update your local Black install as old versions will produce different results.
modified: docs/update_release_note_index.py
modified: geoips/__init__.py
modified: geoips/commandline/ancillary_info/__init__.py
modified: geoips/commandline/geoips_test.py
modified: geoips/commandline/list_available_plugins.py
modified: geoips/commandline/log_setup.py
modified: geoips/create_plugin_registries.py
modified: geoips/data_manipulations/corrections.py
modified: geoips/data_manipulations/merge.py
modified: geoips/dev/output_config.py
modified: geoips/dev/product.py
modified: geoips/image_utils/colormap_utils.py
modified: geoips/image_utils/maps.py
modified: geoips/interfaces/module_based/output_checkers.py
modified: geoips/plugins/modules/algorithms/RGB_Default.py
modified: geoips/plugins/modules/algorithms/absdiff_mst.py
modified: geoips/plugins/modules/algorithms/model/model_channel.py
modified: geoips/plugins/modules/algorithms/model/model_windbarbs.py
modified: geoips/plugins/modules/algorithms/pmw_tb/pmw_37pct.py
modified: geoips/plugins/modules/algorithms/pmw_tb/pmw_89pct.py
modified: geoips/plugins/modules/algorithms/pmw_tb/pmw_color37.py
modified: geoips/plugins/modules/algorithms/pmw_tb/pmw_color89.py
modified: geoips/plugins/modules/algorithms/pressure_winds/windbarbs_dmw.py
modified: geoips/plugins/modules/algorithms/sfc_winds/windbarbs.py
modified: geoips/plugins/modules/algorithms/single_channel.py
modified: geoips/plugins/modules/algorithms/visir/Convective_Storms.py
modified: geoips/plugins/modules/algorithms/visir/Day_Microphys_Summer.py
modified: geoips/plugins/modules/algorithms/visir/Day_Microphys_Winter.py
modified: geoips/plugins/modules/algorithms/visir/Day_Solar.py
modified: geoips/plugins/modules/algorithms/visir/Dust_RGB.py
modified: geoips/plugins/modules/algorithms/visir/HNear_Constant_Contrast.py
modified: geoips/plugins/modules/algorithms/visir/Natural_Color.py
modified: geoips/plugins/modules/algorithms/visir/Night_Microphys.py
modified: geoips/plugins/modules/algorithms/visir/Night_Vis.py
modified: geoips/plugins/modules/algorithms/visir/Night_Vis_GeoIPS1.py
modified: geoips/plugins/modules/algorithms/visir/Night_Vis_IR.py
modified: geoips/plugins/modules/algorithms/visir/Night_Vis_IR_GeoIPS1.py
modified: geoips/plugins/modules/algorithms/visir/Volcanic_Ash.py
modified: geoips/plugins/modules/algorithms/visir/airmass.py
modified: geoips/plugins/modules/algorithms/visir/nasa_dust_rgb.py
modified: geoips/plugins/modules/colormappers/matplotlib_linear_norm.py
modified: geoips/plugins/modules/colormappers/matplotlib_log_norm.py
modified: geoips/plugins/modules/filename_formatters/text_winds_day_fname.py
modified: geoips/plugins/modules/interpolators/utils/boxdefinitions.py
modified: geoips/plugins/modules/output_formatters/histogram_csv.py
modified: geoips/plugins/modules/output_formatters/histogram_netcdf.py
modified: geoips/plugins/modules/output_formatters/netcdf_full.py
modified: geoips/plugins/modules/output_formatters/netcdf_geoips.py
modified: geoips/plugins/modules/output_formatters/windspeed_awips2_formatter.py
modified: geoips/plugins/modules/readers/abi_netcdf.py
modified: geoips/plugins/modules/readers/ami_netcdf.py
modified: geoips/plugins/modules/readers/amsua_mhs_mirs.py
modified: geoips/plugins/modules/readers/amsub_hdf.py
modified: geoips/plugins/modules/readers/ewsg_netcdf.py
modified: geoips/plugins/modules/readers/gfs_grib.py
modified: geoips/plugins/modules/readers/gmi_hdf5.py
modified: geoips/plugins/modules/readers/imerg_hdf5.py
modified: geoips/plugins/modules/readers/modis_hdf4.py
modified: geoips/plugins/modules/readers/seviri_hrit.py
modified: geoips/plugins/modules/readers/ssmi_binary.py
modified: geoips/plugins/modules/readers/ssmis_binary.py
modified: geoips/plugins/modules/readers/viirs_netcdf.py
modified: geoips/plugins/modules/readers/windsat_idr37_binary.py
modified: geoips/plugins/modules/sector_metadata_generators/bdeck_parser.py
modified: geoips/plugins/modules/sector_metadata_generators/fdeck_parser.py
modified: geoips/plugins/modules/sector_metadata_generators/tc_sector_file_parser.py
modified: geoips/pydantic_models/v1/__init__.py
modified: geoips/pydantic_models/v2alpha1/__init__.py
modified: geoips/sector_utils/tc_tracks_database.py
modified: geoips/utils/memusg.py
modified: geoips/xarray_utils/xr_to_dtree.py
modified: setup/download_test_data.py
modified: tests/unit_tests/commandline/cli_top_level_tester.py
modified: tests/unit_tests/commandline/test_get_commandline_instructions.py
modified: tests/unit_tests/plugins/modules/output_checkers/test_output_checkers.py
modified: tests/unit_tests/plugins/test_get_plugin.py
modified: tests/unit_tests/plugins/yaml/test_all_yaml_plugins.py
modified: tests/unit_tests/pydantic_models/v1/test_pydantic_models.py
modified: tests/unit_tests/pydantic_models/v1/utils.py
modified: tests/unit_tests/pydantic_models/v1/workflows/test_workflow_step_definition_model.py
modified: tests/unit_tests/pydantic_models/v1/workflows/test_workflows.py
modified: tests/unit_tests/schema/test_yaml_schema.py
modified: tests/unit_tests/xarray_utils/test_dtree.py
modified: tests/unit_tests_long/plugins/modules/readers/test_readers.py
modified: tests/utils/test_interfaces.py
Enhancement#
Add doc8 linting to code testing framework#
Added doc8 documentation linter support to the check_code.sh. Check_code now supports running doc8 with a maximum line length of 120 characters (our standard) on documentation source files.
modified: tests/utils/check_code.sh
Class based plugins – registry updates#
In an effort to convert all of our module_based plugins to class_based plugins, we need to update multiple areas of the codebase. One large refactor that needs to occur is plugin registry related updates, which is what is handled in this PR.
This update includes a refactor of the BaseModuleInterface to become BaseClassInterface, and updates all module_based interfaces to make use of that new class. Directory changes to come soon.
Additionally, updated create_plugin_registries.py and plugin_registry.py to make use of the new updates to the BaseClassInterface, and added logic to handle the conversion of module_based interfaces to a true plugin class, or the omission of that step if a True class plugin is provided.
Lastly, created a dummy class-based algorithm to test the new plugin registry infrastructure and updated all failing unit/integration tests to match the new functionality added.
added: geoips/interfaces/class_based_plugin.py
added: geoips/plugins/classes/algorithms/single_channel.py
modified: geoips/commandline/geoips_describe.py
modified: geoips/commandline/list_available_plugins.py
modified: geoips/create_plugin_registries.py
modified: geoips/interfaces/__init__.py
modified: geoips/interfaces/base.py
modified: geoips/interfaces/module_based/__init__.py
modified: geoips/interfaces/module_based/algorithms.py
modified: geoips/interfaces/module_based/colormappers.py
modified: geoips/interfaces/module_based/coverage_checkers.py
modified: geoips/interfaces/module_based/databases.py
modified: geoips/interfaces/module_based/filename_formatters.py
modified: geoips/interfaces/module_based/interpolators.py
modified: geoips/interfaces/module_based/output_checkers.py
modified: geoips/interfaces/module_based/output_formatters.py
modified: geoips/interfaces/module_based/procflows.py
modified: geoips/interfaces/module_based/readers.py
modified: geoips/interfaces/module_based/sector_adjusters.py
modified: geoips/interfaces/module_based/sector_metadata_generators.py
modified: geoips/interfaces/module_based/sector_spec_generators.py
modified: geoips/interfaces/module_based/title_formatters.py
modified: geoips/plugin_registry.py
modified: geoips/plugins/modules/procflows/config_based.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: tests/unit_tests/commandline/test_geoips_describe_plugin.py
modified: tests/unit_tests/commandline/test_geoips_list_plugins.py
modified: tests/unit_tests/commandline/test_geoips_list_source_names.py
modified: tests/unit_tests/plugin_registries/files/bad/invalid_interfaces.yaml
modified: tests/unit_tests/plugin_registries/files/bad/missing_lowest_keys.yaml
modified: tests/unit_tests/plugin_registries/files/bad/missing_plugin_types.yaml
modified: tests/unit_tests/plugin_registries/files/good/data_fusion.yaml
modified: tests/unit_tests/plugin_registries/files/good/geoips.yaml
modified: tests/unit_tests/plugin_registries/files/good/geoips_clavrx.yaml
modified: tests/unit_tests/plugin_registries/files/good/geoips_plugin_example.yaml
modified: tests/unit_tests/plugin_registries/files/good/overcast_package.yaml
modified: tests/unit_tests/plugin_registries/files/good/recenter_tc.yaml
modified: tests/unit_tests/plugin_registries/files/good/template_basic_plugin.yaml
modified: tests/unit_tests/plugin_registries/files/good/template_fusion_plugin.yaml
modified: tests/unit_tests/plugin_registries/test_plugin_registries.py
modified: tests/unit_tests/plugins/test_get_plugin.py
modified: tests/unit_tests/plugins/test_get_plugin_metadata.py
modified: tests/utils/test_interfaces.py
renamed: geoips/interfaces/module_based/__init__.py => geoips/interfaces/class_based/__init__.py
renamed: geoips/interfaces/module_based/algorithms.py => geoips/interfaces/class_based/algorithms.py
renamed: geoips/interfaces/module_based/colormappers.py => geoips/interfaces/class_based/colormappers.py
renamed: geoips/interfaces/module_based/coverage_checkers.py => geoips/interfaces/class_based/coverage_checkers.py
renamed: geoips/interfaces/module_based/databases.py => geoips/interfaces/class_based/databases.py
renamed: geoips/interfaces/module_based/filename_formatters.py =>
geoips/interfaces/class_based/filename_formatters.py
renamed: geoips/interfaces/module_based/interpolators.py => geoips/interfaces/class_based/interpolators.py
renamed: geoips/interfaces/module_based/output_checkers.py => geoips/interfaces/class_based/output_checkers.py
renamed: geoips/interfaces/module_based/output_formatters.py => geoips/interfaces/class_based/output_formatters.py
renamed: geoips/interfaces/module_based/procflows.py => geoips/interfaces/class_based/procflows.py
renamed: geoips/interfaces/module_based/readers.py => geoips/interfaces/class_based/readers.py
renamed: geoips/interfaces/module_based/sector_adjusters.py => geoips/interfaces/class_based/sector_adjusters.py
renamed: geoips/interfaces/module_based/sector_metadata_generators.py =>
geoips/interfaces/class_based/sector_metadata_generators.py
renamed: geoips/interfaces/module_based/sector_spec_generators.py =>
geoips/interfaces/class_based/sector_spec_generators.py
renamed: geoips/interfaces/module_based/title_formatters.py => geoips/interfaces/class_based/title_formatters.py
Rewrite of text output checker#
This is a rewrite of the text output checker. It adds QOL updates such as optional inline diff printing, optional rich output and optional updating of comparisons (with manual confirmation that is yes compatible). It adds options to GeoIPS for setting user preferences on whether to enable these new features.
modified: geoips/filenames/base_paths.py
modified: geoips/plugins/modules/output_checkers/text.py
Consolidate yaml-based and module-based unit tests into test_pydanitc_models.py#
Unifies YAML-based and module-based Pydantic unit tests into single location, reducing redundancy and simplifying maintenance.
added: tests/unit_tests/pydantic_models/v1/title_formatters/test_cases_neutral.yaml
modified: tests/unit_tests/pydantic_models/v1/conftest.py
modified: tests/unit_tests/pydantic_models/v1/test_pydantic_models.py
deleted: tests/unit_tests/pydantic_models/v1/title_formatters/conftest.py
deleted: tests/unit_tests/pydantic_models/v1/title_formatters/test_title_formatters.py
Adding an output checker pydantic model#
Creating a Pydantic model for output checkers and unit tests to ensure functionality. The general pydantic test flow was also modified to allow for a test_cases_good.yaml file.
deleted: tests/unit_tests/pydantic_models/v1/feature_annotators/test_cases.yaml
deleted: tests/unit_tests/pydantic_models/v1/sectors/test_cases.yaml
added: docs/source/releases/latest/1204-output-checkers-pydantic-model.yaml
added: pydantic_models/v1/output_checkers.py
added: tests/unit_tests/pydantic_models/v1/feature_annotators/test_cases_good.yaml
added: tests/unit_tests/pydantic_models/v1/gridline_annotators/test_cases_good.yaml
added: tests/unit_tests/pydantic_models/v1/sectors/test_cases_good.yaml
added: tests/unit_tests/pydantic_models/v1/output_checkers/test_cases_bad.yaml
added: tests/unit_tests/pydantic_models/v1/output_checkers/test_cases_good.yaml
added: tests/unit_tests/pydantic_models/v1/output_checkers/test_cases_neutral.yaml
added: tests/unit_tests/pydantic_models/v1/title_formatters/test_cases_good.yaml
modified: tests/unit_tests/pydantic_models/v1/conftest.py
modified: tests/unit_tests/pydantic_models/v1/test_pydantic_models.py
modified: tests/unit_tests/pydantic_models/v1/utils.py
modified: tests/unit_tests/pydantic_models/v1/feature_annotators/test_cases_neutral.yaml
Add no_color environment variable to geoips#
The GeoIPS CLI has a few commands which can produce output that is colored differently than most terminal text. We do this via the ‘colorama’ package, however, we did not have a setting to disable colored terminal output if wanted. We took in consideration what other software packages have implemented and decided to reinforce that standard by implementing a new ‘NO_COLOR’ environment variable. If set to True, then GeoIPS will no longer produce colored output via the terminal. Keep in mind that since this variable commonly used among software packages, setting NO_COLOR to True may impact packages outside the scope of GeoIPS. For example, when this variable is enabled, pytest will no longer color its terminal output.
modified: docs/source/concepts/configuration/index.rst
modified: docs/source/concepts/functionality/command-line/index.rst
modified: docs/source/concepts/index.rst
modified: geoips/commandline/commandline_interface.py
modified: geoips/commandline/geoips_command.py
modified: geoips/commandline/geoips_run.py
modified: geoips/commandline/geoips_tree.py
modified: geoips/filenames/base_paths.py
Order based – dealing with generated and non-registered workflow plugins#
This PR updates how we can call the order based procflow. Instead of only being able to provide a registered workflow plugin name to the commandline alongside other procflow arguments GeoIPS supports, we’ve now added two additional arguments in which we can specify a workflow. Those arguments are ‘–filepath’ and ‘–generated’.
Neither of these arguments represent a ‘registered’ workflow plugin. Generated means we’re passing a literal evalation of a python dictionary (representing a workflow plugin), and filepath means we’re passing a path to an unregistered workflow plugin.
WorkflowPluginModel validation occurs for both of the new arguments and will only send the provided workflows to the order based procflow once validation has passed. Otherwise, appropriate errors will be raised.
Unit tests and commandline help information have been updated to reflect the changes made in this PR.
modified: geoips/commandline/ancillary_info/cmd_instructions.yaml
modified: geoips/commandline/geoips_run.py
modified: geoips/plugins/modules/procflows/order_based.py
modified: geoips/pydantic_models/v1/bases.py
modified: tests/unit_tests/commandline/test_geoips_run.py
Base plugin classes#
This PR builds off of #1235 by implementing base plugin classes for all class-based
interfaces (previously module-based interfaces). Additionally, this PR converts all
output_checker plugins to class-based plugins and refactors the output_checker
interface / base plugin code to remove any reference of plugin.module and make
use of functionality via inheritance instead. There is an additional PR for
recenter_tc (NRLMMD-GEOIPS/recenter_tc#54) which converts
the fdeck output_checker to class-based as well.
Additionally, added documentation describing the purpose of class-based plugins and how they impact the plugin registry. There is also an example of how to convert a module-based plugin to a class-based plugin which also describes how the plugin registry can access the metadata of the true class-based plugin.
Finally, refactored the long unit test test_readers by removing any reference of
plugin.module and updating logic to get all information needed for unit tests
via a hardcoded yaml file rather than having testing logic in the module of specific
reader plugins.
added: docs/dev/class_based_plugins.rst
added: geoips/base_class_plugins/__init__.py
added: geoips/base_class_plugins/algorithms.py
added: geoips/base_class_plugins/colormappers.py
added: geoips/base_class_plugins/coverage_checkers.py
added: geoips/base_class_plugins/databases.py
added: geoips/base_class_plugins/filename_formatters.py
added: geoips/base_class_plugins/interpolators.py
added: geoips/base_class_plugins/output_checkers.py
added: geoips/base_class_plugins/output_formatters.py
added: geoips/base_class_plugins/procflows.py
added: geoips/base_class_plugins/readers.py
added: geoips/base_class_plugins/sector_adjusters.py
added: geoips/base_class_plugins/sector_metadata_adjusters.py
added: geoips/base_class_plugins/sector_spec_generators.py
added: geoips/base_class_plugins/title_formatters.py
added: geoips/plugins/classes/output_checkers/geotiff.py
added: geoips/plugins/classes/output_checkers/image.py
added: geoips/plugins/classes/output_checkers/netcdf.py
added: geoips/plugins/classes/output_checkers/text.py
modified: docs/dev/order_based_procflow.rst
modified: geoips/create_plugin_registries.py
modified: geoips/image_utils/maps.py
modified: geoips/interfaces/__init__.py
modified: geoips/interfaces/base.py
modified: geoips/interfaces/class_based/algorithms.py
modified: geoips/interfaces/class_based/colormappers.py
modified: geoips/interfaces/class_based/coverage_checkers.py
modified: geoips/interfaces/class_based/databases.py
modified: geoips/interfaces/class_based/filename_formatters.py
modified: geoips/interfaces/class_based/interpolators.py
modified: geoips/interfaces/class_based/output_checkers.py
modified: geoips/interfaces/class_based/output_formatters.py
modified: geoips/interfaces/class_based/procflows.py
modified: geoips/interfaces/class_based/readers.py
modified: geoips/interfaces/class_based/sector_adjusters.py
modified: geoips/interfaces/class_based/sector_metadata_generators.py
modified: geoips/interfaces/class_based/sector_spec_generators.py
modified: geoips/interfaces/class_based/title_formatters.py
modified: geoips/interfaces/class_based_plugin.py
modified: geoips/interfaces/yaml_based/gridline_annotators.py
modified: geoips/interfaces/yaml_based/sectors.py
modified: geoips/plugin_registry.py
modified: geoips/plugins/classes/algorithms/single_channel.py
modified: geoips/plugins/modules/procflows/config_based.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: geoips/plugins/modules/readers/ami_netcdf.py
modified: geoips/plugins/modules/readers/amsr2_netcdf.py
modified: geoips/plugins/modules/readers/amsr2_remss_winds_netcdf.py
modified: geoips/plugins/modules/readers/cygnss_netcdf.py
modified: geoips/plugins/modules/readers/gmi_hdf5.py
modified: geoips/plugins/modules/readers/sar_winds_netcdf.py
modified: geoips/plugins/modules/readers/scat_knmi_winds_netcdf.py
modified: geoips/plugins/modules/readers/scat_noaa_winds_netcdf.py
modified: geoips/plugins/modules/readers/smap_remss_winds_netcdf.py
modified: tests/unit_tests/plugins/modules/output_checkers/test_output_checkers.py
modified: tests/unit_tests_long/plugins/modules/readers/reader_dataset_variable_mappings.yaml
modified: tests/unit_tests_long/plugins/modules/readers/test_readers.py
modified: tests/utils/test_interfaces.py
deleted: geoips/plugins/modules/output_checkers/geotiff.py
deleted: geoips/plugins/modules/output_checkers/image.py
deleted: geoips/plugins/modules/output_checkers/netcdf.py
deleted: geoips/plugins/modules/output_checkers/text.py
Obp pydantic product / product default models#
Adds new pydantic models used to validate product and product_default plugins. These updates are intended to completely replace the JSON schema we currently use to validate yaml-based plugins. Minor changes were needed to be made to geoips/pydantic/bases.py to take into account the differences in product plugins.
Additionally, this update adds unit testing for these new models, and updated the .gitignore file to ignore both types of our plugin registry files (.json, .yaml).
added: geoips/pydantic/products.py
added: tests/unit_tests/pydantic/product_defaults/test_cases_bad.yaml
added: tests/unit_tests/pydantic/product_defaults/test_cases_neutral.yaml
added: tests/unit_tests/pydantic/products/test_cases_bad.yaml
added: tests/unit_tests/pydantic/products/test_cases_neutral.yaml
modified: .gitignore
modified: geoips/dev/product.py
modified: geoips/interfaces/yaml_based/gridline_annotators.py
modified: geoips/interfaces/yaml_based/product_defaults.py
modified: geoips/interfaces/yaml_based/products.py
modified: geoips/interfaces/yaml_based/sectors.py
modified: geoips/plugin_registry.py
modified: geoips/pydantic/bases.py
modified: tests/unit_tests/plugins/yaml/sectors/test_sectors_jsonschema.py
modified: tests/unit_tests/pydantic/test_pydantic_models.py
modified: tests/unit_tests/pydantic/utils.py
Add output to geoips config delete-registries#
Currently the command ‘geoips config delete-registries’ prints nothing useful. It just prints None, whether or not the command successfully deleted one or more registry files. I’ve updated this command to produce some useful output, which determines whether or not a file was deleted, and where it can be found.
modified: geoips/commandline/geoips_config.py
modified: geoips/plugin_registry.py
Realtime#
Update config_based procflow to pull sector_adjuster_kwargs from output config yaml#
Supporting the use of sector_adjuster_kwargs was previously only supported by the single_source procflow via command line arguments. The config_based procflow was updated to pull sector_adjuster_kwargs from the output_config YAML, and defaults to an empty dictionary if none are specified in the output_config.
geoips#1038
modified: geoips/plugins/modules/procflows/config_based.py
Testing#
Test data abi dataset update v1.17.1#
This PR updates the URL used for the test_data_abi dataset. It now points to V1.17.1 instead of our V1.15.1 dataset. This is a simple one line change to the URL of that dataset that is used via the CLI for test dataset installation.
modified: geoips/commandline/ancillary_info/test-data-urls.yaml
Cli unit tests – test dataset name updates#
We have unit tests for every GeoIPS CLI command and the test module for ‘geoips list test-datasets’ needs updated. This is a simple update to the hardcoded values for our test datasets. We now retrieve the appropriate list of test dataset names in the same manner we do for the actual CLI command.
modified: tests/unit_tests/commandline/cli_top_level_tester.py
Allow importing validation and lint test lists from geoips repo, and add validation pytest markers#
Add repopath and pkgname to geoips repo setup_environment, rather than just geoips_repopath and geoips_pkgname. Note these get overwritten in individual plugin repositories after calling geoips setup_environment.
Update lint test list to use repopath and pkgname rather than geoips_repopath and geoips_pkgname, so the same test list can be imported by all the plugin repos.
Add validation test list, using the current repopath and pkgname as arguments to list plugins and test interfaces calls.
Add validation pytest marker to pytest.ini
modified: pytest.ini
modified: tests/integration_tests/test_reponame.py
modified: tests/utils/test_integration.py
Creating tiny sector for testing#
Add new global_200km_longlat sector for integration testing.
geoips#1033 - low res global tiny sector
added: geoips/plugins/yaml/sectors/integration_tests/global_200km_longlat.yaml
Have fail on missing data default to true#
Rather than having to update every plugin repo integration test script to add the required fail_on_missing_data flag, just have it default to True in the main run bash script function in the main geoips repo. If the flag is supported in a plugin repo, it will be passed in appropriately, if it is not supported in a plugin repo, it will just default to True and the plugin repo integration tests won’t break.
Also print the value of fail_on_missing_data to terminal.
geoips#1029 - 1.18.0 updates
modified: tests/integration_tests/test_integration.py
Allow disabling dumping all failed log contents to terminal#
Set GEOIPS_TEST_SUPPRESS_PYTEST_FAILED_LOG_CONTENTS env var to True to avoid printing all failed log output from integration tests to terminal.
When running interactively might be nice to suppress, if running exhaustively don’t suppress and all log output will be contained in a single log file.
geoips#1029 - 1.18.0 updates
modified: tests/integration_tests/test_integration.py
modified: geoips/filenames/base_paths.py
Consolidated env variable naming for all geoips_test env vars#
Prepend all test related environment variables with GEOIPS_TEST.
GEOIPS_TEST_OUTPUT_CHECKER_THRESHOLD_IMAGE 0.05
GEOIPS_TEST_PRINT_TEXT_OUTPUT_CHECKER_TO_CONSOLE True
GEOIPS_TEST_SUPPRESS_PYTEST_FAILED_LOG_CONTENTS False
GEOIPS_TEST_SECTOR_CREATE_ANNOTATED_OUTPUTS False
GEOIPS_TEST_SECTOR_CREATE_GEOTIFF_OUTPUTS False
GEOIPS_TEST_PROMPT_TO_OVERWRITE_COMPARISON_FILE_IF_MISMATCH False
geoips#1029 - 1.18.0 updates
modified: tests/integration_tests/test_integration.py
modified: geoips/filenames/base_paths.py
modified: tests/integration_tests/tiny_sectors/tiny_sectors_geostationary.sh
modified: geoips/plugins/classes/output_checkers/image.py
modified: geoips/plugins/classes/output_checkers/text.py
Temporarily disable doc8 by default for check_code.sh#
Until all rst docs are updated, disable the doc8 check in check_code.sh so the tests pass.
geoips#1029 - 1.18.0 updates
modified: tests/utils/check_code.sh
Remove print statements in unit tests#
To clean up pytest output, remove extraneous print statements from unit tests.
geoips#1029 - 1.18.0 updates
modified: tests/unit_tests/pydantic_models/v1/test_pydantic_models.py
modified: tests/unit_tests/pydantic_models/v1/utils.py
Feature#
Order-based procflow: add title formatters#
Introduce Pydantic model defining and validating OBP-v1 TitleFormatterArgumentsModel
added: geoips/pydantic_models/v1/title_formatters.py
added: tests/unit_tests/pydantic_models/v1/title_formatters/conftest.py
added: tests/unit_tests/pydantic_models/v1/title_formatters/test_cases_bad.yaml
added: tests/unit_tests/pydantic_models/v1/title_formatters/test_title_formatters.py
Order-based procflow: add coverage checkers#
Define OBP Pydantic model for Coverage Checker plugins.
added: docs/source/releases/latest/1180-OBP-add-coverage-checkers-pydantic-models.yaml
added: geoips/pydantic_models/v1/coverage_checkers.py
added: geoips/plugins/yaml/workflows/read_test_v1_coverage_checker.yaml
added: tests/unit_tests/pydantic_models/v1/coverage_checkers/conftest.py
added: tests/unit_tests/pydantic_models/v1/coverage_checkers/test_cases_bad.yaml
added: tests/unit_tests/pydantic_models/v1/coverage_checkers/test_coverage_checkers.py
modified: geoips/plugin_registry.py
modified: geoips/plugins/modules/coverage_checkers/center_radius.py
modified: geoips/plugins/modules/coverage_checkers/center_radius_rgba.py
modified: geoips/plugins/modules/coverage_checkers/masked_arrays.py
modified: geoips/plugins/modules/coverage_checkers/numpy_arrays_nan.py
modified: geoips/plugins/modules/coverage_checkers/rgba.py
modified: geoips/plugins/modules/coverage_checkers/windbarbs.py
modified: geoips/plugins/modules/procflows/order_based.py
modified: geoips/plugins/yaml/workflows/read_test_v1.yaml
modified: geoips/pydantic_models/v1/workflows.py
Plotting#
Histogram colorbar and static sector conformity.#
Created a histogram colorbar that helps visulize distribution of values for annotated imagery. Fixed static sectors that had non-standard formatting, this includes not having the region sub-key, or having spacing in the metadata which would lead to spacing in the output filepath. Added a north american static sector.
added: tests/scripts/amsr2.tc.89H-Physical.imagery_annotated_colorhist.sh
added: geoips/plugins/yaml/sectors/static/north_america.yaml
modified: geoips/image_utils/mpl_utils.py
modified: geoips/plugins/modules/output_formatters/imagery_annotated.py
modified: geoips/plugins/yaml/sectors/static/e_pacific.yaml
modified: geoips/plugins/yaml/sectors/static/guam.yaml
modified: geoips/plugins/yaml/sectors/static/hawaii.yaml
modified: geoips/plugins/yaml/sectors/static/rio_de_janeiro.yaml
modified: geoips/plugins/yaml/sectors/static/w_atlantic.yaml
Documentation#
Fixed docstrings on sgli reader and on hncc algorithm.#
Added better description of source on High Near Constant Contrast algorithm, attributing paper the algorithm was derived from. Fixed several copy paste bugs within the SGLI reader.
modified: geoips/plugins/modules/readers/sgli_l1b_hdf5.py
modified: geoips/plugins/modules/algorithms/visir/HNear_Constant_Contrast.py
Add release note for version 1.18.0#
Release note on release branch to force tag/release, package/publish, and deploy docs.
Additionally add any updated files from geoips_ci/sync
geoips#1029 - 1.18.0 updates
added: docs/source/releases/latest/1.18.0-release-note.yaml
modified: geoips_ci/sync updated files
Update upcoming and tagged versions#
Tagged version: 1.18.0
Upcoming version: 1.18.1
geoips#1029 - 1.18.0 updates
modified: .github/versions/tagged_version
modified: .github/versions/upcoming_version
Temporary bug fix#
Temporary fix for incorrect solar and satellite azimuth angles#
This is a temporary bug fix for satellite and solar angle calculations. Proper and longterm fix will be to update the backend calculations.
There is a bug in the satellite azimuth calculation where it is off by 180° It doesn’t seem like it is an issue with the output being 0-360 vs -180 - 180. Converting to -180 - 180 will not yield the correct result.
There is another bug in the solar azimuth calculation that causes the output to be in radians, and in 0-360. If the full disk solar azimuth angle array has minimum greater than zero, then we are dealing with the bugged calculation. Need to apply the following corrections - 1 - Convert from radians to degrees 2 - Convert from 0-360 to -180 - 180
geoips#1045
modified: geoips/plugins/modules/readers/utils/geostationary_geolocation.py
Minor bug fix#
Add storm_id to fields returned by f-deck parser#
Import assemble_invest_storm_id and assemble_numbered_storm_id functions from the bdeck parser to create the storm_id. The storm_id is now also returned by the fdeck parser.
geoips#1035
modified: geoips/plugins/modules/sector_metadata_generators/fdeck_parser.py
Bug fix#
Fix plugin registry rebuild check#
This fixes a typo in a check for if GEOIPS_REBUILD_REGISTRIES is set.
modified: geoips/plugin_registry.py
Setup#
Remove geoips_vers#
Only support GEOIPS_VERSION, remove GEOIPS_VERS support. This is mostly unused, but ensure the supported version variable is GEOIPS_VERSION.
geoips#1027
added: setup/config_geoips
added: geoips/filenames/base_paths.py
added: geoips/plugins/modules/procflows/config_based.py
Resource usage statistics#
Add option to write statistics for a config_based procflow write to a json file#
Update the print_mem_usg and checkpoint_usage_stats methods in the PidLog class to write statistics to a JSON file if write_to_json is True.
Add new –write_stats_to_json commandline argument, which is used in config_based, and passed to print_mem_usg and checkpoint_usage_stats at the end of the procflow.
Also improve logic when attempting to determine which type of sectors were processed. Should now be capable of determining if procflow processed dynamic TC sectors, static sectors, or a mix of both.
Add new test script for writing overall resource usage statistics JSON file, and the checkpoint resource usage statistics JSON file. Due to the dynamic nature of these files, we do not attempt to compare to an expected output. This script only makes sure the procflow does not catastrophically fail if these outputs are enabled.
geoips#1027
added: tests/scripts/abi.config_based_output_low_memory_resource_usage_logging.sh
added: tests/yaml_configs/abi_test_low_memory_resource_usage_logging.yaml
modified: geoips/commandline/args.py
modified: geoips/plugins/modules/procflows/config_based.py
modified: geoips/utils/memusg/memusg_tracker.py
Move all memusg utilities to utils/memusg#
Since we now have internal trackers, external process trackers, json outputs, and diagnostic plotting utilities, place them all in a separate utils/memusg package.
Update all imports accordingly for PidLog (readers are updates in the comments only, they do not import directly). Note ahi_hsd reader also imports print_mem_usage directly, so that includes an import update.
geoips#1027
moved: geoips/utils/memusg/memtrack.sh
moved: geoips/utils/memusg/memusg_tracker.py
modified: geoips/plugins/modules/procflows/config_based.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: docstring only geoips/plugins/modules/readers/abi_netcdf.py
modified: import print_mem_usage geoips/plugins/modules/readers/ahi_hsd.py
modified: docstring only geoips/plugins/modules/readers/ami_netcdf.py
modified: docstring only geoips/plugins/modules/readers/ewsg_netcdf.py
modified: docstring only geoips/plugins/modules/readers/seviri_hrit.py
Add json based resource usage plotting capabilities#
Allow plotting resource usage statistics, as enabled through procflow config files.
This relies on json resource usage output files being enabled through the procflow config, then a separate call to plot_json_files to plot the results.
Note this plotting capability was initially set up for only static sector processing. The test script includes both TC and static, which completes successfully, except the resulting checkpoint usage plot includes only the TC but not the static sector. This will be fixed in a future PR to properly track all sector types.
geoips#1027
added: geoips/utils/memusg/boxplots.py
added: geoips/utils/memusg/dataframe_utils.py
added: geoips/utils/memusg/plot_json_files.py