Version 1.14.0a0 (2024-08-16)#

Bug fix: Mac Image Match Bug

Bug fix: Update build_docs.sh to auto-generate <version>.rst with brassy

Bug fix: Add pip install of plugin package to build_docs.sh

Bug fix: Minor bug fixes and updates to AWS download script and abi_l2_netcdf reader

Bug fix: Remove exact comparison failures from image output checker plugin

Bug fix: Add support for algorithm application for xarray dict based families

Bug fix: Fix typo causing product family xrdict_area_product_to_outlist failure

Bug fix: Remove interactive log from plugin unit test

Bug fix: Revert docs/source/releases/index.rst back after build_docs.sh

Bug fix: Fix install on contemporary Debian machines (Ubuntu, etc.)

Bug fix: CLI Bug Fixes and Formatting Updates

Bug fix: Install packages prior to attempting to build their documentation

Bug fix: Ensure return value correctly captured when importing package in build_docs.sh

Bug fix: Fix memusg.py bugs that occur on Mac

Continuous_integration: Change release note .yaml folder from version to latest

Enhancement: New geoips tree command

Continuous integration: Fix CI issues with incorrectly passing/failing release notes

Continuous integration: Disable check_new_release_note on main

Continuous integration: Fix the workflow to run on all PRs

Continuous integration: Auto-PR built release note on new tag

Bug-fix: Fix duplicate logs, cartopy version, and expose unit test

Bug-fix: Correct typo causing product family xrdict_area_product_to_outlist failure

Bug-fix: Fix failing test data CLI unit tests

Installation: Remove references to template_fusion_plugin from full install and test

Installation: Add base and full pip and mamba environment files

Documentation: Add pinkrst and brassy to dependencies

Documentation: Update LICENSE and DISTRIBUTION to latest approved version

Documentation: Add top level descriptions to minor releases

Documentation: Integrate new docs into old docs for workshop

Documentation: Write System Requirements

Documentation: Updated CLI Documentation to reflect the current state of the CLI

Ci: Added brassy to CI

Testing: Call pytest_long from check_code.sh via all_test_data vs all

Testing: Add NOAA-20, NOAA-21, and NPP downloads to NOAA AWS download script

Refactoring updates: Applied Scope to CLI Arguments

Refactoring updates: Change geoips get to geoips describe

Refactoring updates: Refactored geoips list interface to use a class factory

Refactoring updates: Refactored geoips get <interface_name> commands to use a class factory

Refactoring updates: Merged create_sector_image into the CLI

Bug fixes: Resolve algorithm_colormapper typo in single_source procflow

Enchancement: Add VIIRS HDF5 SDR reader

Regression fix: Fix Regression of geoips tree command

Bug fix#

Mac Image Match Bug#

While testing Mac for the 2024 tutorial, it was found quite a few tests were failing due to bad image comparisons. Upon greater inspection, it was found that these tests were failing only due to a couple of different pixels, which we assume is due to different backend mechanics depending on your OS. Instead of failing a test if a single pixel is outside of a certain threshold, we’ve decided to test an image’s match with another pixel via a group basis.

Ie. If num_pixels_mismatched_by_threshold / total_num_pixels > x%, then we’ll raise an error. This is a more lenient test, but will still catch differences between two images. All in all, we wan’t to ignore tiny differences between images and catch the more glaring differences that affect lots of pixels when they occur.

modified: geoips/plugins/modules/output_checkers/image.py

Update build_docs.sh to auto-generate <version>.rst with brassy#

Update build_docs.sh to call brassy to auto-generate the release notes prior to doc build.

Also from doc build, call new script to update the release note index to add a “latest” section if there are latest YAML release notes. We may want to use this script to add versioned sections to index.rst in the future as well, but for now it only auto-adds latest.

modified: .gitignore
modified: docs/build_docs.sh
modified: docs/source/releases/index.rst
added: docs/update_release_note_index.py
deleted: docs/source/releases/latest.rst

Add pip install of plugin package to build_docs.sh#

Add pip install of plugin package to build_docs.sh to ensure package is installed before attempting to build API docs.

Currently, build_docs.sh checks is the current package is installed, and if not it pip installs it. In the future will likely just error, and assume the package is installed already if attempting to build documentation.

modified: docs/build_docs.sh

Minor bug fixes and updates to AWS download script and abi_l2_netcdf reader#

From GEOIPS/geoips#666: 2024-07-16, Final Bug Fixes Prior to Workshop Minor bug fixes to abi_l2_netcdf reader for masking out bad lat/lon points, and to download_noaa_aws.sh script that allow the script to download other collections from AWS (i.e. L2 data)

modified: geoips/plugins/modules/readers/abi_l2_netcdf.py
modified: tests/download_noaa_aws.sh

Remove exact comparison failures from image output checker plugin#

From GEOIPS/geoips#666: 2024-07-16, Final Bug Fixes Prior to Workshop

Primary functional change

Only check for percentage pixel diffs with the thresholded call to pixelmatch
Only include exact check for purposes of reporting and outputing exact diff image.

A few additional updates to help figure out the best way forward for comparison

Return immediately if the arrays are identical (avoid running pixelmatch)
Ensure we run exact and thresholded pixelmatch if the arrays are not identical
Output separate exact and thresholded diff images

modified: geoips/plugins/modules/output_checkers/image.py

Add support for algorithm application for xarray dict based families#

From GEOIPS/geoips#666: 2024-07-16, Final Bug Fixes Prior to Workshop

Ensure all xarray dict based product families are supported within the single source procflow, including sectored and unsectored dictionaries of xarrays, algorithm and no algorithm application, and with and without area defs. Note ALL are not supported, but better identified what is missing / supported now.

modified: geoips/plugins/modules/procflows/single_source.py

Fix typo causing product family xrdict_area_product_to_outlist failure#

From GEOIPS/geoips#666: 2024-07-16, Final Bug Fixes Prior to Workshop

Product families that did not have pre-defined output lists and expected data passed unmodified from the readers to the output formatters were failing due to a missing else statement ensuring the list of final_products was set even when the output file list was not pre-defined. Add else statement to set final_products to curr_products when output file list not defined.

This resolved an error with product family xrdict_area_product_to_outlist when it was requested without sectoring, but it worked if sectoring was requested (followed a different route through the code for each).

modified: geoips/plugins/modules/procflows/single_source.py

Remove interactive log from plugin unit test#

From GEOIPS/geoips#666: 2024-07-30, Final Bug Fixes Prior to Workshop

This test script unnecessarily requires test data. It was updated to skip the test if $GEOIPS_TESTDATA_DIR doesn’t exist, but then it would still fail if some test data was available, but not the test data required for the abi reader. Comment this test out altogether for now - we should create a new one at some point that does not require test data if we want to test the interactive logs from a directly imported plugin.

modified: tests/unit_tests/commandline/test_log_setup.py

Revert docs/source/releases/index.rst back after build_docs.sh#

Revert index.rst back to original contents after build_docs.sh completes. Since the intent is for index.rst to always be auto updated, and never manually edited, we will ensure all changes are reverted after build_docs completes.

modified: docs/build_docs.sh

Fix install on contemporary Debian machines (Ubuntu, etc.)#

When installing GeoIPS on a machine with only python3 (and not python) we get WARNING: 'python --version' failed, please install python >= 3.9 before proceeding because the code looks for python explicitly. Some Linux distributions decided during the transition from Python 2 to Python 3 that python should always refer to Python 2, and the command to run Python 3 would be python3 with a 3 at the end. Debian and Ubuntu did this. This change adds a message to users informing them how they can change their system so calls to python are re-routed to their local python3.

modified: setup/check_system_requirements.sh

CLI Bug Fixes and Formatting Updates#

This PR fixed small bugs and formatting inconsistencies with the CLI. Specifically, this PR addressed the format of the output of ‘geoips describe’ commands, to make the keys in the yaml output all be in title format rather than (some_val:) vs. (Some Val:). In this PR we were working on getting CLI documentation up to date, and discovered a bug with ‘geoips list packages’ where the ‘–columns’ flag was not actually working. This was a simple fix and that functionality is now working as expected. Unit tests have been updated to address these changes.

modified: geoips/commandline/commandline_interface.py
modified: geoips/commandline/geoips_command.py
modified: geoips/commandline/geoips_describe.py
modified: geoips/commandline/geoips_list.py
modified: tests/unit_tests/commandline/test_geoips_describe_interface.py
modified: tests/unit_tests/commandline/test_geoips_plugin.py
modified: tests/unit_tests/commandline/test_geoips_list_packages.py

Install packages prior to attempting to build their documentation#

Add a step in the build_sphinx_html CI job to install the package for which we want to build documentation prior to building the documentation. Also remove the installation code from the build_docs.sh script.

This fixes an issue where installation doesn’t always happen correctly causing the docs to fail to build.

modified: docs/build_docs.sh
modified: .github/workflows/doc-lint-test.yaml

Ensure return value correctly captured when importing package in build_docs.sh#

Ensure we capture return value from import mypkgname correctly when determining if plugin package is installed. Previously always failed, so always attempted to reinstall the plugin package.

modified: docs/build_docs.sh

Fix memusg.py bugs that occur on Mac#

This branch was needed as there was recently a bug introduced on Mac that would prevent any use of GeoIPS. This was introduced in geoips/utils/memusg.py, where a Process method was used that was only available on Linux, FreeBSD, SunOS. The method was Process.cpu_num and was used in tracking information about the CPU Count used during a certain GeoIPS Process. Unfortunately, this is unavailable on Mac and caused bugs when trying to run GeoIPS.

To fix this, we added a platform.system() == Linux check for Process.cpu_num calls which ensured that function would only be ran if on a Linux system. Otherwise skip that call and don’t collect inforamation on that variable.

modified: geoips/utils/memusg.py

Continuous_integration#

Change release note .yaml folder from version to latest#

Moved the yaml release files from docs/source/release/v(version number)/* to docs/source/release/latest/*, which now builds to latest.rst. Added latest.rst because brassy does not automatically build .rst files at the moment. Ideally, brassy would create latest.rst, which could be pulled down and built into the docs locally. Without latest.rst the docs will not build. So adding a blank file serves as a placeholder until the CI automatically builds and commits a latest.rst file. Added latest to /docs/source/release/index.rst so docs build. The release note not edited check SHOULD NOT PASS, because…. it was edited 😄

added: docs/source/releases/latest.rst
added: docs/source/releases/latest/687-change-release-note-yaml-folder-from-version-to-latest.yaml
modified: .github/workflows/doc-lint-test.yaml
modified: docs/source/releases/index.rst

Enhancement#

New `geoips tree` command#

From GEOIPS#627: 2024-05-31, Add functionality to the CLI which prints out a tree of available commands

The GeoIPS CLI provides a variety of commands which aren’t necessarily easily exposed via geoips -h. To improve this issue, we’ve added a geoips tree command which exposes all GeoIPS CLI commands in a tree-like fashion. This way, we can expose all commands that are available via the GeoIPS CLI, and expose the depth in which these commands exist.

By displaying the commands in a depthwise structure, users can understand what commands are available and how they are called.

If you just call geoips tree, you’ll get the full command tree in a non-colored, verbose output.

The output of running geoips tree is shown below.

geoips tree

geoips
    geoips config
        geoips config install
    geoips get
        geoips get family
        geoips get interface
        geoips get package
        geoips get plugin
    geoips list
        geoips list interface
        geoips list interfaces
        geoips list packages
        geoips list plugins
        geoips list scripts
        geoips list test-datasets
        geoips list unit-tests
    geoips run
        geoips run single_source
        geoips run data_fusion
        geoips run config_based
    geoips test
        geoips test linting
        geoips test script
    geoips tree
    geoips validate

geoips tree additionaly provides optional arguments to filter the output of this command. Shown below are these optional arguments and descriptions of what each argument does.

--color
- The output of geoips tree might be a little hard to interpret. If you want the output of geoips tree to be colored by depth, make sure to use the --color flag. (Defaults to False)
--max-depth
- How many levels of the tree we’d like to expose. Defaults to two levels, which is shown above.
--short-name
- The output of geoips tree provides the full command string at each level. If you just want the literal command name and every level, make sure to provide this flag. (Defaults to False)

added: geoips/commandline/geoips_tree.py
added: tests/unit_tests/commandline/test_geoips_tree.py
modified: docs/source/userguide/command_line.rst
modified: geoips/commandline/ancillary_info/cmd_instructions.yaml
modified: geoips/commandline/commandline_interface.py
modified: geoips/commandline/geoips_command.py
modified: geoips/filenames/base_paths.py
modified: tests/unit_tests/commandline/cli_top_level_tester.py

Continuous integration#

Fix CI issues with incorrectly passing/failing release notes#

Changed boolean check for whether or not release notes have been added to fix bug where the check for nesessary yaml files would pass unexpectedly. Additionally, changed check to prevent manual editing of release files to fail on changes to any *.rst files in docs/source/release/ instead of just latest.rst in the same path.

modified: .github/workflows/doc-lint-test.yaml

Disable check_new_release_note on `main`#

Disable check_new_release_note on main because it compares against main… and thus always fails.

modified: .github/workflows/doc-lint-test.yaml

Fix the workflow to run on all PRs#

Update the workflow to run tests regardless of which branch a PR points to. Previously had only run if pointing to main.

modified: .github/workflows/doc-lint-test.yaml

Auto-PR built release note on new tag#

Added workflow to build, commit and PR a built release note with brassy

added: .github/workflows/new-brassy-note.yaml

Bug-fix#

Fix duplicate logs, cartopy version, and expose unit test#

From GEOIPS#685: 2024-07-17, Fix unit test that will fail if any plugin packages with console scripts are installed

There are a few lingering bugs that need fixed before the workshop starts in August. These were relatively simple to fix so I merged three bug fixes into this PR.

Bug #1 managed to stay in GeoIPS for a while, and resulted in duplicate log statements for every log level. This was caused by geoips.commandline.log_setup:setup_logging and would result in LOG X (num times setup_logging called) per a program’s execution. To fix this, I added two global variables in setup_logging, once of which tests if that function has already been called, the second being a log variable that will be returned the first time, and every time after this function is called. With this update, we need to make sure that the lowest log-level requested should be called FIRST. Otherwise, it will be obfuscated by higher level log levels.

Bug #2 was a simple fix, and was just a version change for Cartopy in pyproject.toml. There was a minor pixel difference in some tests outputs using version 0.22.0, so we updated this to gt=0.23.0.

Bug #3 was a possible bug that was introduced by the expose command. There was a hardcoded portion of a unit test for that command that could result in failed tests if certain packages had console scripts. We’ve refactored this command to be dynamic, and now should pass no matter what packages are provided.

Bug #4 was related to commandline instructions unit tests, specifically the tests that checked whether or not a file was newer than another file. These work locally, but git causes problems with the files write time and these tests sometimes fail. Since we are confident in the functionality of this code, we’ve decided to remove these unit tests for the time being.

modified: geoips/commandline/commandline_interface.py
modified: geoips/commandline/log_setup.py
modified: pyproject.toml
modified: tests/unit_tests/commandline/test_expose.py
modified: tests/unit_tests/commandline/test_get_commandline_instructions.py
deleted: tests/unit_tests/commandline/cmd_instructions/json_newer/*
deleted: tests/unit_tests/commandline/cmd_instructions/yaml_newer/*

Correct typo causing product family xrdict_area_product_to_outlist failure#

Correct typo causing product family xrdict_area_product_to_outlist failure. Product families that did not have pre-defined output lists and expected data passed unmodified from the readers to the output formatters were failing due to a missing else statement ensuring the list of final_products was set even when the output file list was not pre-defined. Add else statement to set final_products to curr_products when output file list not defined. This resolved an error with product family xrdict_area_product_to_outlist when it was requested without sectoring, but it worked if sectoring was requested (followed a different route through the code for each).

modified: geoips/plugins/modules/procflows/single_source.py

Fix failing test data CLI unit tests#

Fix failing test data CLI unit tests. test_geoips_config_install.py and test_log_setup.py had 1 or more tests that were failing, which caused the CI to fail as well. Fix these unit tests so we can actually use the CI to address problems that are coming from new PRs.

modified: tests/unit_tests/commandline/test_log_setup.py
modified: tests/unit_tests/commandline/cli_top_level_tester.py
deleted: tests/unit_tests/commandline/test_geoips_config_install.py

Installation#

Remove references to template_fusion_plugin from full install and test#

From GEOIPS#666: 2024-07-17, Final bug fixes prior to workshop

Remove all references to template_fusion_plugin. No longer supporting template_fusion_plugin - only template_basic_plugin with very basic plugins, and geoips_plugin_example with more extensive examples.

modified: tests/integration_tests/full_install.sh
modified: tests/integration_tests/full_test.sh

Add base and full pip and mamba environment files#

From GEOIPS#666: 2024-07-17, Final bug fixes prior to workshop

Add base and full environment dumps from version 1.13.0.

modified: environments/mamba_base_package_list_1.13.0_20240713.yml
modified: environments/mamba_full_package_list_1.13.0_20240717.yml
modified: environments/pip_base_requirements_1.13.0_20240713.txt
modified: environments/pip_full_requirements_1.13.0_20240717.txt

Documentation#

Add pinkrst and brassy to dependencies#

Add pinkrst and brassy to dependencies.

modified: pyproject.toml

Update LICENSE and DISTRIBUTION to latest approved version#

Update LICENSE and DISTRIBUTION to latest approved version.

modified: LICENSE
modified: DISTRIBUTION

Add top level descriptions to minor releases#

Added summaries to the minor releases in the docs to make searching for a specific change easier.

modified: docs/source/releases/index.rst

Integrate new docs into old docs for workshop#

Removed old files, tidied up prose and integrated new docs into old docs via links/toctrees.

added: docs/source/license/index.rst
added: docs/source/new-docs/contribute/adding-functionality.rst
added: docs/source/new-docs/contribute/code-of-conduct.rst
added: docs/source/new-docs/contribute/coding_standards.rst
added: docs/source/new-docs/contribute/git-github.rst
added: docs/source/releases/latest/integrate-new-docs-into-old-docs.yaml
modified: docs/source/_templates/index_PKG.html
modified: docs/source/contact/aboutus.rst
modified: docs/source/contact/index.rst
modified: docs/source/devguide/contributors.rst
modified: docs/source/devguide/dev_setup.rst
modified: docs/source/devguide/documentation_strategy.rst
modified: docs/source/devguide/git_workflow.rst
modified: docs/source/devguide/index.rst
modified: docs/source/devguide/software_requirements_specification.rst
modified: docs/source/devguide/unit_tests.rst
modified: docs/source/devguide/xarray_standards.rst
modified: docs/source/geoips_api/index.rst
modified: docs/source/introduction/description_geoips.rst
modified: docs/source/introduction/index.rst
modified: docs/source/new-docs/concepts/functionality/index.rst
modified: docs/source/new-docs/concepts/scope/index.rst
modified: docs/source/new-docs/contact.rst
modified: docs/source/new-docs/homepage.rst
modified: docs/source/starter/expert_installation.rst
modified: docs/source/starter/extending.rst
modified: docs/source/starter/index.rst
modified: docs/source/starter/installation.rst
modified: docs/source/starter/mac_installation.rst
modified: docs/source/starter/starter_examples.rst
modified: docs/source/starter/windows_installation.rst
modified: docs/source/userguide/command_line.rst
modified: docs/source/userguide/geoips_structure.rst
modified: docs/source/userguide/index.rst
modified: docs/source/userguide/plugin_development/algorithm.rst
modified: docs/source/userguide/plugin_development/colormapper.rst
modified: docs/source/userguide/plugin_development/feature_annotator.rst
modified: docs/source/userguide/plugin_development/gridline_annotator.rst
modified: docs/source/userguide/plugin_development/output_formatter.rst
modified: docs/source/userguide/plugin_development/product.rst
modified: docs/source/userguide/plugin_development/product_default.rst
modified: docs/source/userguide/plugin_development/reader.rst
modified: docs/source/userguide/plugin_development/static_sector.rst
modified: docs/source/userguide/plugin_extend.rst
modified: docs/source/userguide/plugin_registries.rst
deleted: docs/dev/coding_standards.rst
deleted: docs/source/devguide/build_docs.rst
deleted: docs/source/introduction/conduct.rst
deleted: docs/source/introduction/examples_output.rst
deleted: docs/source/introduction/function_summary.rst
deleted: docs/source/new-docs/contribute/adding-functionality/index.rst
deleted: docs/source/new-docs/contribute/code-of-conduct/index.rst
deleted: docs/source/new-docs/contribute/coding-standards/git-github.rst
deleted: docs/source/userguide/function_list.rst

Write System Requirements#

Added system requirements to the documentation. This includes minimum and recommended requirements for users and developers as well as language limiting the applicability of the system requirements.

modified: docs/source/new-docs/getting-started/system-requirements/index.rst

Updated CLI Documentation to reflect the current state of the CLI#

This PR updated the CLI documentation to reflect the current state of the CLI code. Largely, this PR went through the documentation of the CLI commands and made sure that what was documented matched what would actually happen when that command was ran. There was a duplicate entry in the documentation that was removed as well.

modified: docs/source/userguide/commandl_line.rst

Ci#

Added brassy to CI#

Added brassy builds to the CI

modified: .github/workflows/doc-lint-test.yaml

Testing#

Call pytest_long from check_code.sh via all_test_data vs all#

Make pytest_long called via “all_test_data” vs “all”

modified: tests/utils/check_code.sh

Add NOAA-20, NOAA-21, and NPP downloads to NOAA AWS download script#

Add NOAA-20, NOAA-21, and NPP downloads to NOAA AWS download script

modified: tests/download_noaa_aws.sh

Refactoring updates#

Applied Scope to CLI Arguments#

From GEOIPS#637: 2024-06-06, Using parser.parse_known_args in the CLI to apply scope to arguments

While this PR ended up not making use of parse_known_args (It was buggy and resulted in overly complex conditionals), we did end up finding a way to apply scope (ie. share arguments from parents to children) to CLI commands to reduce repeated portions of the code. This is also nice because we only have to make code changes to one place if we want to alter arguments that are shared by various commands.

To do this, we created a ParentParsers Object in geoips.commandline.geoips_command which contains argument parsers that add arguments which will be shared by some, if not all of the child command classes. For example, the geoips_parser attribute of ParentParsers will be shared to all child commands. This is because every command is a child of geoips (the name of GeoipsCLI class). A similar ideology applies to list_parser, except that only children of list, ie. GeoipsListPackages, GeoipsListPlugins, etc. will get the arguments created by list_parser. This allows for us to share universal arguments such as log level and command specific argumetns such as package_name.

modified: geoips/commandline/commandline_interface.py
modified: geoips/commandline/geoips_command.py
modified: tests/unit_tests/commandline/cli_top_level_tester.py

Change geoips get to geoips describe#

From GEOIPS#495: 2024-04-15, Change CLI Command ‘geoips get’ to ‘geoips describe’

This update renames all references of ‘geoips get’ to ‘geoips describe’. ‘Describe’ is a better name for this command as it generally provides additional information about a certain GeoIPS artifact, whereas ‘get’ could be a bit ambiguous as users may think we are actually getting a certain plugin, interface, etc. Another reason for this update is that all references to the CLI in the tutorial use ‘geoips describe’, rather than ‘geoips get’. This PR was only a nomenclature change and did not impact any of the actual functionality of the CLI.

added: geoips/commandline/geoips_describe.py
added: tests/unit_tests/commandline/test_geoips_describe_family.py
added: tests/unit_tests/commandline/test_geoips_describe_interface.py
added: tests/unit_tests/commandline/test_geoips_describe_package.py
added: tests/unit_tests/commandline/test_geoips_describe_plugin.py
deleted: geoips/commandline/geoips_get.py
deleted: tests/unit_tests/commandline/test_geoips_get_family.py
deleted: tests/unit_tests/commandline/test_geoips_get_interface.py
deleted: tests/unit_tests/commandline/test_geoips_get_package.py
deleted: tests/unit_tests/commandline/test_geoips_get_plugin.py
modified: docs/source/userguide/command_line.rst
modified: geoips/commandline/ancillary_info/alias_mapping.yaml
modified: geoips/commandline/ancillary_info/cmd_instructions.yaml
modified: geoips/commandline/commandline_interface.py
modified: geoips/commandline/geoips_command.py

Refactored `geoips list interface` to use a class factory#

From GEOIPS#576: 2024-05-11, CLI: Reduce number of command layers where possible and appropriate From GEOIPS#573: 2024-05-11, Use class factories to generate subcommands for commands like ``geoips get family`` and ``geoips get interface``

While this PR doesn’t fix all of the problems addressed in the issues above, it is our initial attempt at creating a class factory for certain CLI commands to reduce the verbage needed to execute those commands. Spefically in this branch, we’ve addressed the command class GeoipsListSingleInterface. We now use the aforementioned class as a base class to build GeoipsListSingleInterface<interface_name> classes at runtime.

By doing so, we can now run geoips list <interface_name> rather than geoips list interface <interface_name>. This improves the readability of this command and makes it much easier to type via the CLI. We expect in future PRs to address similar commands, such as geoips get family <family_name> and geoips get plugin <interface_name> <plugin_name> for the reasons mentioned previously.

modified: docs/source/userguide/command_line.rst
modified: geoips/commandline/ancillary_info/cmd_instructions.yaml
modified: geoips/commandline/geoips_command.py
modified: geoips/commandline/geoips_list.py
modified: tests/unit_tests/commandline/cli_top_level_tester.py
modified: tests/unit_tests/commandline/test_geoips_list_interface.py

Refactored `geoips get <interface_name>` commands to use a class factory#

From GEOIPS#576: 2024-05-11, CLI: Reduce number of command layers where possible and appropriate From GEOIPS#573: 2024-05-11, Use class factories to generate subcommands for commands like ``geoips get family`` and ``geoips get interface``

This branch refactors commands which use the form of geoips get <interface_name> to make use of class factories. Before, we had to run commands such as:

geoips get interface <interface_name>
geoips get family <interface_name> <family_name>
geoips get plugin <interface_name> <family_name>

The aforementioned commands were overly verbose and needed some refactoring to reduce the levels required to execute a certain command. In this branch, we refactored GeoipsGetInterface to be a base command for get <interface_name>, get <interface_name> family <family_name>, and get <interface_name> <plugin_name>. This resulted in the removal of GeoipsGetFamily and GeoipsGetPlugin and the refactoring of GeoipsGetInterface to include the functionality of those other commands. We can use class factories for each interface to execute these commands now.

On top of this, we implemented aliases for a variety of CLI commands. For an exact listing of aliases available for each command, see geoips/commandline/ancillary_info/alias_mapping.yaml, which includes aliases supported for each command. Now, we can run a verbose command such as:

geoips list algorithms -p geoips

and replace it with

geoips ls algs -p geoips

This also reduces the verbosity of certain CLI commands. Note, the unit tests have been modified to reflect these changes.

modified: .gitignore
modified: pyproject.toml
modified: docs/source/_templates/conf_PKG.py
modified: docs/source/userguide/command_line.rst
modified: geoips/commandline/ancillary_info/cmd_instructions.yaml
modified: geoips/commandline/cmd_instructions.py
modified: geoips/commandline/commandline_interface.py
modified: geoips/commandline/geoips_command.py
modified: geoips/commandline/geoips_get.py
modified: tests/unit_tests/commandline/cli_top_level_tester.py
modified: tests/unit_tests/commandline/test_geoips_get_family.py
modified: tests/unit_tests/commandline/test_geoips_get_interface.py
modified: tests/unit_tests/commandline/test_geoips_get_package.py
modified: tests/unit_tests/commandline/test_geoips_get_plugin.py
modified: tests/unit_tests/commandline/test_geoips_list_interface.py
modified: tests/unit_tests/commandline/test_geoips_list_interfaces.py
modified: tests/unit_tests/commandline/test_geoips_list_packages.py
modified: tests/unit_tests/commandline/test_geoips_list_plugins.py
modified: tests/unit_tests/commandline/test_geoips_list_scripts.py
modified: tests/unit_tests/commandline/test_geoips_list_test_datasets.py
modified: tests/unit_tests/commandline/test_geoips_list_unit_tests.py
modified: tests/unit_tests/commandline/test_get_commandline_instructions.py
added: geoips/commandline/ancillary_info/alias_mapping.yaml

Merged `create_sector_image` into the CLI#

From GEOIPS#636: 2024-06-05, Merge create_sector_image.py functionality into the CLI

This update moves the functionality from the independent console script create_sector_image onto the CLI under the command geoips test sector. Since the CLI has been added to main, we are slowly consolidating all independent console scripts onto the CLI, so we have a uniform was of executing console commands with GeoIPS.

There was a slight modification to this command, as we now only allow one sector to be provided to this command instead of a list of sectors. This fits better with the nomenclature of the command and can be ran multiple times if the user wants to create multiple sector images.

Unit tests have been added for this command as well.

added: tests/unit_tests/commandline/test_geoips_test_sector.py
added: docs/source/images/command_line_examples/canada.png
deleted: geoips/commandline/create_sector_image.py
modified: geoips/commandline/ancillary_info/cmd_instructions.yaml
modified: geoips/commandline/geoips_test.py
modified: pyproject.toml
modified: tests/scripts/console_script_create_sector_image.sh
modified: docs/source/userguide/command_line.rst
modified: docs/source/userguide/plugin_development/static_sector.rst

Bug fixes#

Resolve algorithm_colormapper typo in single_source procflow#

Resolve algorithm_interpolator_colormapper typo in single_source procflow Pass in xarray.Dataset() rather than alg_xarray as the “interp_xarray” to the “perform_interpolation” function. Previously the xarray was not being interpolated, because it thought the alg_xarray was the pre-interpolated array. This bug was introduced when refactoring single source to avoid re-interpolating. We should probably add an integration test for this product family.

modified: geoips/plugins/modules/procflows/single_source.py

Enchancement#

Add VIIRS HDF5 SDR reader#

Created reader for VIIRS HDF5 SDR data, correcting for bowtie distortion for single and multiple files.

added: geoips/plugins/modules/readers/viirs_sdr_hdf5.py

Regression fix#

Fix Regression of geoips tree command#

With the addition of Command Class Factories, Shared Arguments, and Aliases, we had an expected regression that would impact the functionality of the command ‘geoips tree’. While we still may make updates to this command in the future, regarding its output and how we’d like to unit test it, we now have working in a much cleaner fashion. This PR addressed removing duplicate entries in the output of the tree, as well hardcoded some conditionals for corner cases that would cause duplicate entries do to the structure of aliases. For example, we had to add conditionals when we encountered ‘sector’ as a command name, as this is an alias for ‘geoips list sectors’ and ‘geoips describe sectors’, and is the actual command name of ‘geoips test sector’.

Additionally, we commented out the functionality of ‘hyperlinking’ the text as the current documentation is not being built up on GitHub and renders that functionality useless at the moment. In the future, we expect to bring the hyperlink functionality back in and update how we retrieve the command name as we change how ‘prog’ is set for each commands’ subparser attribute.

modified: geoips/commandline/ancillary_info/alias_mapping.yaml
modified: geoips/commandline/geoips_describe.py
modified: geoips/commandline/geoips_list.py
modified: geoips/commandline/geoips_tree.py

Version 1.14.0a0 (2024-08-16)#

Bug fix#

Mac Image Match Bug#

Update build_docs.sh to auto-generate <version>.rst with brassy#

Add pip install of plugin package to build_docs.sh#

Minor bug fixes and updates to AWS download script and abi_l2_netcdf reader#

Remove exact comparison failures from image output checker plugin#

Add support for algorithm application for xarray dict based families#

Fix typo causing product family xrdict_area_product_to_outlist failure#

Remove interactive log from plugin unit test#

Revert docs/source/releases/index.rst back after build_docs.sh#

Fix install on contemporary Debian machines (Ubuntu, etc.)#

CLI Bug Fixes and Formatting Updates#

Install packages prior to attempting to build their documentation#

Ensure return value correctly captured when importing package in build_docs.sh#

Fix memusg.py bugs that occur on Mac#

Continuous_integration#

Change release note .yaml folder from version to latest#

Enhancement#

New geoips tree command#

Continuous integration#

Fix CI issues with incorrectly passing/failing release notes#

Disable check_new_release_note on main#

Fix the workflow to run on all PRs#

Auto-PR built release note on new tag#

Bug-fix#

Fix duplicate logs, cartopy version, and expose unit test#

Correct typo causing product family xrdict_area_product_to_outlist failure#

Fix failing test data CLI unit tests#

Installation#

Remove references to template_fusion_plugin from full install and test#

Add base and full pip and mamba environment files#

Documentation#

Add pinkrst and brassy to dependencies#

Update LICENSE and DISTRIBUTION to latest approved version#

Add top level descriptions to minor releases#

Integrate new docs into old docs for workshop#

Write System Requirements#

Updated CLI Documentation to reflect the current state of the CLI#

Ci#

Added brassy to CI#

Testing#

Call pytest_long from check_code.sh via all_test_data vs all#

Add NOAA-20, NOAA-21, and NPP downloads to NOAA AWS download script#

Refactoring updates#

Applied Scope to CLI Arguments#

Change geoips get to geoips describe#

Refactored geoips list interface to use a class factory#

Refactored geoips get <interface_name> commands to use a class factory#

Merged create_sector_image into the CLI#

Bug fixes#

Resolve algorithm_colormapper typo in single_source procflow#

Enchancement#

Add VIIRS HDF5 SDR reader#

Regression fix#

Fix Regression of geoips tree command#

This Page

New `geoips tree` command#

Disable check_new_release_note on `main`#

Refactored `geoips list interface` to use a class factory#

Refactored `geoips get <interface_name>` commands to use a class factory#

Merged `create_sector_image` into the CLI#