Version 1.14.5 (2025-04-30)#

  • Enhancement: Switch default vimrc colorscheme to a light theme, default

  • Enhancement: Add noaa-21 support to viirs_netcdf reader

  • Enhancement: Add missingrequiredxarraymetadata error

  • Enhancement: Allow setting log format strings via base_paths/config

  • Enhancement: Pass config_dict to database writer

  • Enhancement: Support scale and offset information in cogeotiff output formatter

  • Enhancement: Add geotiff_imagery_path to base_paths and use in geotiff_fname

  • Enhancement: Add imagery paths to geoips_utils replace_geoips_paths

  • Enhancement: Add support for ews-g3 (goes-14)

  • Enhancement: Allow lat/lon based area_defs to create correct resolution in filenames.

  • Bug fix: Xfail if unit-test directory does not exist for given package.

  • Bug fix: Allow command line tests for editable and non-editable installs

  • Bug fix: Check if unit-tests folder exists when determining if it exists.

  • Bug fix: Close abi netcdf file object after reading

  • Bug fix: Replace amsu-b references with mhs, and update mirs netcdf reader

  • Bug fix: Replace all geoips paths in output file list, not just geoips_outdirs

  • Bug fix: Fix clobbering when passing product_database_writer_kwargs

  • Bug fix: Resolve ami geolocation bugs.

  • Bug fix: Resolve issues masking near terminator for seviri reader.

  • Bug fix: Fix bug preventing invest b-deck files from being added to decks database

  • Bug fix: Fix which version of geoips is shown at the command line

  • Bug fix: Black and flake8 updates for cogeotiff

  • Bug fix: Return error code from build_docs.sh

  • Bug fix: Roll back xarray, attempt fix knmi scat time conversion xarray bug

  • Bug fix: Revert naming of mhs variables

  • Bug fix: Abi netcdf non-resampled read roi and sat zen cutoff same as resampled

  • Bug fix: Add support for adding b-deck files to tc tracks database

  • Documentation: Update upcoming and tagged versions

  • Documentation: Add more informative error messages for create_plugin_registries

  • Continuous integration: Build stable image on tag.

  • Refactor: Consolidate setup config environment variables

  • Refactor: Consolidate geotiff output formatters

  • Testing: Update geotiff test outputs

Enhancement#

Switch default vimrc colorscheme to a light theme, default#

Use light by default, not dark

modified: setup/bash_setup/vimrc_ale

Enhancement#

Add noaa-21 support to viirs_netcdf reader#

JPSS-2 -> NOAA-21 platform name in viirs_netcdf.

modified: geoips/plugins/modules/readers/viirs_netcdf.py

Enhancement#

Add missingrequiredxarraymetadata error#

Error to raise if at any point during geoips processing an xarray object does not contain all required metadata. We should add a function to validate the output of readers, etc, and raise this error if any metadata is missing.

modified: geoips/errors.py

Enhancement#

Allow setting log format strings via base_paths/config#

Move GEOIPS_LOGGING_LEVEL to geoips/filenames/base_paths.py (which will eventually be the geoips config), and add GEOIPS_LOGGING_DATFMT_STRING and GEOIPS_LOGGING_FMT_STRING for making the logging output format fully configurable. Maintain the same defaults, and allow passing in an explicit string now rather than using the “verbose” keyword arg (which was a bool, and mapped to one of 2 format strings).

Now just default to None in args, so the defaults are pulled from gpaths in log_setup.py directly.

modified: geoips/commandline/log_setup.py
modified: geoips/filenames/base_paths.py
modified: geoips/commandline/args.py

Enhancement#

Pass config_dict to database writer#

To ensure all processing workflow configuration information is available to the database writers, just pass the full config_dict. When order basd procflow is implemented, I expect to be able to pass the full order based procflow config into the database writers as well, and we will be able to adjust to get the appropriate information out at that time.

modified: geoips/plugins/modules/procflows/config_based.py

Enhancement#

Support scale and offset information in cogeotiff output formatter#

Ensure scale and offset information is appropriately included in GeoTIFF output to support obtaining physical values from the resulting COG. This will allow creating 8 bit GeoTIFFs, and still retrieving actual reflectances, brightness temperatures, etc from the GeoTIFF.

Update cogeotiff.py to retrieve the appropriate scaling information from the product YAMLs, and write it to the GeoTIFF.

modified: geoips/plugins/modules/output_formatters/cogeotiff.py

Enhancement#

Add geotiff_imagery_path to base_paths and use in geotiff_fname#

Previously geotiffs were going to ANNOTATED_IMAGERY

modified: geoips/filenames/base_paths.py
modified: geoips/plugins/modules/filename_formatters/geotiff_fname.py

Enhancement#

Add imagery paths to geoips_utils replace_geoips_paths#

This includes WWW and internal imagery paths specified in base_paths

modified: geoips/geoips_utils.py

Enhancement#

Add support for ews-g3 (goes-14)#

Add hooks for EWS-G3 (GOES-14) to ewsg reader.

modified: geoips/plugins/modules/readers/ewsg_netcdf.py

Enhancement#

Allow lat/lon based area_defs to create correct resolution in filenames.#

I don’t think this actually works 100%. May need to adjust slightly.

modified: geoips/plugins/modules/filename_formatters/geoips_fname.py

Bug fix#

Xfail if unit-test directory does not exist for given package.#

Rather than failing the unit tests overall if a single GeoIPS plugin package does not contain a unit-test directory, xfail. Perhaps in the future we want to 100% require unit-test directories, but for now we should not cause the full code check to fail due to a single package that does not contain unit tests.

modified: tests/unit_tests/commandline/test_geoips_list_unit_tests.py

Bug fix#

Allow command line tests for editable and non-editable installs#

Previously non-editable installs would fail the commandline-based unit tests. Ensure editable and non-editable installs both pass. GitHub Actions workflows should be updated to test both editable and non-editable installs, as appropriate.

Note LOG.warnings were added to the geoips_list_unit_tests unit test when the unit-test directory does not exist, in addition to the xfail. Currently we are planning to xfail when unit-test directory does not exist.

modified: tests/unit_tests/commandline/cli_top_level_tester.py
modified: tests/unit_tests/commandline/test_geoips_list_scripts.py
modified: tests/unit_tests/commandline/test_geoips_list_unit_tests.py

Bug fix#

Check if unit-tests folder exists when determining if it exists.#

Previously this checked if the length of the list of package_names was 1 to determine if the unit-tests folder existed. I don’t think that is correct. Instead, check if os.path.exists on the unit-tests folder, and if it does not exist, raise an error that the unit-tests folder does not exist.

modified: geoips/commandline/geoips_list.py

Bug fix#

Close abi netcdf file object after reading#

ABI netcdf files were opened for reading, but never closed. Ignoring the terrible practice, while this may have worked with netcdf4<1.7.0, this likely has been causing issues with config based processing using netcdf4>1.7.0. In certain config based procflows too many netcdf file objects were left open after reading in the requested data, causing Python to crash. This error was very difficult to track down, and this explanation is only a hunch based off this error that only popped up once out of many, many tests - > 19_224022 abi_netcdf.py 720 INFO: Only need metadata from first file, returning > There are 835 HDF5 objects open! > Report: open objects on 72057594037927942 > 28322 Segmentation fault (core dumped)

No segfaults have been observed after closing the netcdf file, so thinking this was the source of the error.

modified: geoips/plugins/modules/readers/abi_netcdf.py

Bug fix#

Replace amsu-b references with mhs, and update mirs netcdf reader#

The amsub_mirs reader was a bit misleading. AMSU-B has not been around since 2005, and has been replaced with the Microwave Humidity Sounder (MHS). The MIRS files read in by the amsub_mirs reader actually contains both AMSU-A and MHS data, but the reader only read in the MHS channels. In addition to updating this reader to ingest the AMSU-A data, we made the decision to replace all references of AMSU-B with MHS. This was also reflected in the product YAMLs, and therefor the test scripts as well.

The amsub_mirs reader was renamed to amsua_mhs_mirs to better reflect which data are being ingested, and sets the top-level metadata source name to amsu-a_mhs. To make this all the more complicated, in order to have the correct source name listed in the file name and title, the reader will change the source name to either amsu-a or mhs at the dataset level. i.e. The source name in METADATA.attrs will always be amsu-a_mhs, AMSUA.attrs will be amsu-a, and MHS.attrs will be mhs. Due to this switching, we had to add amsu-a_mhs as a source input to all amsu-a and mhs product yamls.

There was no problem with implementing the varying dataset source name in the single source procflow, but it did cause issues with the config based procflow. The config based procflow relies on checking the top level METADATA to get the product plugin. To solve this problem, we added the ability to loop through all datasets and check each source name. This was only a problem in two locations, and while the solution implemented in this pull request works this funtionality should really be toggled (e.g. if METADATA.source_name == multi)

An mhs product YAML previously existed, but it was not actually used. This PR renames the amsu-b product YAML as mhs, and replaces all the contents of the old mhs product YAML.

Also add a new amsu-a product - currently only supports the RainRate product.

deleted: geoips/plugins/yaml/products/amsu-b.yaml
deleted:
tests/outputs/amsub_mirs.tc.183-3H.imagery_annotated/20210419_235400_WP022021_amsu-b_metop-a_183-3H_115kts_100p00_1p0.png
moved: geoips/plugins/modules/readers/amsb_mirs.py -> geoips/plugins/modules/readers/amsua_mhs_mirs.py
added: geoips/plugins/yaml/products/amsu-a.yaml
added:
tests/outputs/mhs_mirs.tc.183-3H.imagery_annotated/20210419_235400_WP022021_mhs_metop-a_183-3H_115kts_100p00_1p0.png
added:
tests/outputs/amsua_mirs.tc.RainRate.imagery_annotated/20210419_235400_WP022021_amsu-a_metop-a_RainRate_115kts_32p68_1p0.png
added:
tests/outputs/amsua_mirs.tc.RainRate.imagery_annotated/20210419_235400_WP022021_amsu-a_metop-a_RainRate_115kts_32p68_1p0.png.yaml
added: tests/scripts/amsua_mirs.tc.rainrate.imagery_annotated.sh
modified: geoips/plugins/modules/readers/amsua_mhs_mirs.py
modified: geoips/plugins/yaml/products/mhs.yaml
modified:
tests/outputs/amsub_mirs.tc.183-3H.imagery_annotated/20210419_235400_WP022021_mhs_metop-a_183-3H_115kts_100p00_1p0.png.yaml
modified: tests/scripts/mhs_mirs.tc.183-3H.imagery_annotated.sh

Bug fix#

Replace all geoips paths in output file list, not just geoips_outdirs#

Use existing GeoIPS utility, replace_geoips_paths, in order to ensure all paths are replaced in the output file list, and not just GEOIPS_OUTDIRS.

Also, ensure the same env vars are replaced in check_output_file_list.sh as are replaced in replace_geoips_paths (not just GEOIPS_OUTDIRS)

modified: geoips/plugins/modules/procflows/single_source.py
modified: geoips/plugins/modules/procflows/config_based.py
modified: tests/utils/check_output_file_list.sh

Bug fix#

Fix clobbering when passing product_database_writer_kwargs#

Fix bug that clobbered the available_sectors settings for a given sector when passing the –product_database_writer_kwargs command line argument to the config_based procflow. Previously when passing –product_database_writer_kwargs, the only information saved under available_sectors -> <SECTOR> would be the information provided by –product_database_writer_kwargs. This fix now overrides any existing settings, rather than offering a bulk replacement.

modified: geoips/plugins/modules/procflows/config_based.py

Bug fix#

Resolve ami geolocation bugs.#

Fixed related bugs to masking data near terminator, and AMI geolocation bugs.

modified: geoips/plugins/modules/readers/ami_netcdf.py

Bug fix#

Resolve issues masking near terminator for seviri reader.#

Fixed related bugs to masking data near terminator, and AMI geolocation bugs.

modified: geoips/plugins/modules/readers/seviri_hrit.py

Bug fix#

Fix bug preventing invest b-deck files from being added to decks database#

Fix bug that was preventing b-deck invest files from being added to the TC decks database. The regular expression check for b-deck files was only supporting named storm files and not invests, which contain the storm start time in the deck file name.

Also add error handling if either a b-deck or g-deck file cannot be parsed and added to the decks database.

modified: geoips/sector_utils/tc_tracks_database.py

Bug fix#

Fix which version of geoips is shown at the command line#

Filter out the deploy tags so the latest tagged version of GeoIPS is shown.

modified: setup/config_geoips

Bug fix#

Black and flake8 updates for cogeotiff#

Black and flake8 formatting ONLY for cogeotiff.py

modified: geoips/plugins/modules/output_formatters/cogeotiff.py

Bug fix#

Return error code from build_docs.sh#

Avoid incorrect 0 return from build_docs.sh if build_docs.py fails.

modified: docs/build_docs.sh

Bug fix#

Roll back xarray, attempt fix knmi scat time conversion xarray bug#

This didn’t quite work, checking it in for reference, roll back xr.

modified: pyproject.toml
modified: geoips/plugins/modules/readers/scat_knmi_winds_netcdf.py

Bug fix#

Revert naming of mhs variables#

Revert naming of MHS variables in mhs product config and amsua_mhs_mirs reader. Having the variables named as MHS_Chan<N>_AT instead of Chan<N>_AT was causing issues with other plugins, such as recenter_tc. We should keep the old naming style for now, at least.

modified: geoips/plugins/modules/readers/amsua_mhs_mirs.py
modified: geoips/plugins/yaml/products/mhs.yaml

Bug fix#

Abi netcdf non-resampled read roi and sat zen cutoff same as resampled#

Previously there was no sat zenith cut off for abi non-resampled reads, and ROI was not applied consistently with resampled

update non-resampled to use ROI factor of 5 to match resampled reads.

update sat zenith cutoff for non-resampled reads to 75 to match resampled.

modified: geoips/plugins/modules/readers/abi_netcdf.py

Bug fix#

Add support for adding b-deck files to tc tracks database#

The update_tc_tracks_database previously only supported G-Deck files, which are not available outside NRL. This commandline utility/database should support tracking B-Deck files. The files are largely similar, biggest difference relates to the index of the storm name between the two files, and how lat/lon values are stored in the files.

modified: geoips/sector_utils/tc_tracks_database.py

Documentation#

Update upcoming and tagged versions#

  • Tagged version: 1.14.5

  • Upcoming version: 1.15.1

modified: .github/versions/tagged_version
modified: .github/versions/upcoming_version

Documentation#

Add more informative error messages for create_plugin_registries#

Ensure informative error message when docstring does not exist in plugins.

modified: geoips/create_plugin_registries.py

Continuous integration#

Build stable image on tag.#

Ensure stable geoips docker image is built on tag/release. This ensures the latest stable image is available for plugin packages to use in PR status checks.

added: .github/workflows/build-stable-image.yaml

Refactor#

Consolidate setup config environment variables#

  • Add create_brassy_changelog helper function

  • Use $GEOIPS_CONFIG_PATH and $GEOIPS_CONFIG_FILE env vars for reference

  • Use $GEOIPS_SYSTEM_VERSION if defined in prompt, else $GEOIPS_VERSION

modified: setup/config_geoips
modified: setup/bash_setup/color_prompt

Refactor#

Consolidate geotiff output formatters#

Have cogeotiff_rgba and geotiff_standard both call cogeotiff. Eventually probably delete these, unless we find there is a benefit to having the ability to explicitly request no-cog (geotiff_standard), or explicitly request RGBA (auto-detected in cogeotiff)

modified: geoips/plugins/modules/output_formatters/cogeotiff_rgba.py
modified: geoips/plugins/modules/output_formatters/geotiff_standard.py

Testing#

Update geotiff test outputs#

Ensure correct outputs for geotiff test scripts after all the geotiff updates.

modified: tests/outputs/ami.tc.WV.geotiff/20231208_030032_SH032024_ami_GK-2A_WV_115kts_100p00_1p0.tif
modified:
tests/outputs/viirsday.global.Night-Vis-IR.cogeotiff_rgba/20210209.074210.noaa-20.viirs.Night-Vis-IR.global.2p00.NASA.20p0.tif
modified: tests/outputs/ahi.tc.WV.geotiff/20200405_000000_SH252020_ahi_himawari-8_WV_100kts_100p00_1p0.tif
modified:
tests/outputs/amsr2.global_20km_longlat.89H-Physical.cogeotiff/20200518.062048.gcom-w1.amsr2.89H-Physical.global_20km_longlat.18p37.star.20p0.tif
On this page

This Page