Distribution Statement
# # # This source code is subject to the license referenced at
# # # NRLMMD-GEOIPS.

Version 1.12.1 (2024-02-16)#

  • Bug fixes

    • Add output_kwargs to xarray_data family output formatter

    • Bug fix in interp_grid scipy.griddata interpolator wrapper

    • Allow overriding interpolator arguments correctly when using product_defaults

    • Make DATABASESUCCESS log.interactive

    • Add longitude wrapping for self-registered data

    • Update SAR reader to handle changed attribute information in file metadata

    • Add longitude wrapping to overpass predictor

    • Update geoips_fname filename formatter to include output_dict kwarg

  • Refactoring updates

    • Remove compare outputs module command line argument

  • Enhancements

    • histogram_netcdf output formatter

    • Add new –product_db_writer_kwargs command line argument

    • Add GeoKOMPSAT reader

    • Add support for adding multiple overpass prediction points/observers for large sectors

    • Add new kwarg to sector create_test_plot to return figure, axis, and map objects

    • Add support for creating product composites in config_based processing

    • Add general RGB product defaults.

  • Installation updates

    • Add create_plugin_registries after base_install.sh

    • Update from Python 3.9 to 3.10 in setup.sh

    • Update installation.rst to use standard installation process

    • Update check_system_requirements to support pulling test data from github

    • Update check_system_requirements to include -v for pip installs

    • Update check_system_requirements to allow GEOIPS_MODIFIED_BRANCH

    • Update check_system_requirements to allow dumping pip and mamba envs

    • Add check_system_requirements function to run arbitrary commands

    • Add check_system_requirements functions to check command and check env var

    • Add check_system_requirements function to install mamba package

    • Raise error when “interface” not defined on plugin during create_plugin_registries

    • Update pyproject.toml dynamic versioning to strict=false

  • GitHub Actions Updates

    • Finalize HTML/pdf build for PR validation

    • Allow calling build_docs.sh without GEOIPS_PACKAGES_DIR set

    • Add GitHub pages documentation deployment workflow

  • Documentation Updates

    • Update sphinx conf.py to support pdf build

    • Require updating release note directly rather than CHANGELOG.rst

    • Add unit testing documentation

    • Ignore */lib/* directories for sphinx api doc

    • Added comments indicating what “interface = None” means

    • Added comments indicating why F401 must be ignored in interfaces/__init__.py

  • Testing Updates

    • Update reader unit tests to support multiple tests, and comparing mean

    • Fail on realtime test if copy file dir does not exist

    • Single pixel difference update in ABI Visible global test output

    • Single pixel difference update in AHI WV TC GEOTIFF test output

  • Add release note for v1.12.1

Refactoring Updates#

Remove compare outputs module command line argument#

This is no longer used, since the output_checkers are defined via a plugin interface. Remove all references to this in all procflows/command line args.

modified: geoips/commandline/args.py
modified: geoips/plugins/modules/procflows/single_source.py
modified: geoips/plugins/modules/procflows/config_based.py

Bug Fixes#

Add output_kwargs to xarray_data family output formatter#

Ensure unsupported kwargs are stripped.

modified:   geoips/plugins/modules/procflows/single_source.py

Update geoips_fname filename formatter to include output_dict kwarg#

To conform with standard filename formatter calls, include “output_dict” kwarg in geoips_fname filename formatter

modified: geoips/plugins/modules/filename_formatters/geoips_fname.py

Ensure center lon between -180 and 180 in overpass predictor#

Longitudes must be between -180 and 180 when creating ephem sector observer for multiple prediction points in large sectors.

modified: geoips/sector_utils/overpass_predictor.py

Make DATABASESUCCESS log.interactive#

modified:   geoips/plugins/modules/procflows/config_based.py
modified:   geoips/plugins/modules/procflows/single_source.py

Allow overriding interpolator arguments correctly when using product_defaults#

From GEOIPS#450: 2023-12-21, Allow overriding interpolator kwargs

Update calls to interpolator within single_source procflow to remove unsupported kwargs. This ensures kwargs that exist in the plugin that is used for product_defaults do not break calls to the plugin used in the product if the kwargs in the call signature are not the same.

modified: geoips/plugins/modules/procflows/config_based.py
modified: geoips/plugins/modules/procflows/single_source.py

Bug fix in interp_grid scipy.griddata interpolator wrapper#

From GEOIPS#448: 2024-01-12, interp_grid bug fix

Fix typo: correctly set numx/y grid to width/height vs pixel_size_x/y.

modified:   geoips/plugins/modules/interpolators/scipy_wrappers/interp_grid.py

Add longitude wrapping for self-registered data#

From GEOIPS#456: 2024-01-26, Fix dating cutting off at date line

Wrap longitudes for the self-registered area definition to the -180 to 180 range so that data is not cut off at the date lline.

modified: geoips/plugins/modules/procflows/single_source.py

Update SAR reader to handle changed attribute information in file metadata#

From GEOIPS#463: 2024-02-09, handle change in SAR file metadata

A change in the SAR metadata in the SAR files was observed around Jan 2024, where “SAR” was no longer in the wind_xarray.source attribute. Current logic requires “SAR” to be in both the wind_xarray.source and wind_xarray.title attributes. The change to the wind_xarray.source attribute lead to the reader not ingesting the wind data from SAR files. The change in the metadata is likely a bug, but should be handled appropriately. SAR Reader now checks in “SAR” is in either the wind_xarray.source OR wind_xarray.title attributes.

  • wind_xarray.source returns "BLANK", which is likely a bug in upstream production.

modified:   geoips/plugins/modules/readers/sar_winds_netcdf.py

Enhancements#

histogram_netcdf output formatter#

Outputs histogram of values for individual image.

new file:   geoips/plugins/modules/output_formatters/histogram_netcdf.py
new file:   tests/scripts/amsr2.tc.89H-Physical.histogram_netcdf.sh

Add general RGB product defaults#

From GEOIPS#468: 2024-02-14, create generic RGB product/algorithm

Replace existing RGB-based product_defaults with a single “RGB_Default.yaml” product defaults. Update viirs.yaml to point to that product_defaults, but override the algorithm to point to Night-Vis-IR or Night-Vis-IR-GeoIPS1.

Eventually, we may want to support a generic “RGB” algorithm that allows specifying the actual individual red green and blue guns from within the YAML file (by passing in arguments / steps), but for now this at least removes one duplicate file.

deleted:  product_defaults/visir/Night-Vis-IR.yaml
deleted:  product_defaults/visir/Night-Vis-IR-GeoIPS1.yaml
new:      product_defaults/RGB_Default.yaml
modified: products/viirs.yaml

Add new –product_db_writer_kwargs command line argument#

From GEOIPS#433: 2023-12-21, Add product db writer kwargs

Add new command line argument for specifying product database writer kwargs. These kwargs are passed to the product database writer plugin calls in the single_source procflow. Writer kwargs for config_based procflows will only be controlled in the YAML.

modified: geoips/commandline/args.py
modified: geoips/plugins/modules/procflows/single_source.py

Add GeoKOMPSAT reader#

Add reader for GeoKOMPSAT-2A AMI instrument. Also includes test scripts and comparison images. Also includes unit test functions for metadata.

new file: geoips/plugins/modules/readers/geokompsat_netcdf.py
new file: geoips/plugins/yaml/products/geokompsat.yaml
new file: geoips/plugins/yaml/products/geokompsat.yaml
new file: tests/outputs/geokompsat.static.Infrared.imagery_annotated/20231208.030032.GK-2A.geokompsat.Infrared.geokompsat.45p56.nmsc.10p0.png
new file: tests/outputs/geokompsat.static.Visible.imagery_annotated/20231208.030032.GK-2A.geokompsat.Visible.geokompsat.45p56.nmsc.10p0.png
new file: tests/outputs/geokompsat.tc.WV.geotiff/20231208_030032_SH032024_geokompsat_GK-2A_WV_115kts_100p00_1p0.tif
new file: tests/outputs/geokompsat.WV-Upper.unprojected_image/20231208.030032.GK-2A.geokompsat.WV-Upper.self_register.71p36.nmsc.2p0.png
new file: tests/scripts/geokompsat.static.Infrared.imagery_annotated.sh
new file: tests/scripts/geokompsat.static.Visible.imagery_annotated.sh
new file: tests/scripts/geokompsat.tc.WV.geotiff.sh
new file: tests/scripts/geokompsat.WV-Upper.unprojected_image.sh
modified: docs/source/releases/v1_12_1.rst
modified: tests/test_all.sh

Add support for adding multiple overpass prediction points/observers for large sectors#

Update overpass predictor utility to automatically add multiple predction points (observers) for large sectors. The check_midpoints kwarg was replaced with the new observer_spacing kwarg, which specifies the spacing (degrees) between observer points in the sector. If lat/lon extent of the domain exceeds the specified spacing, multiple observer are automatically added across the sector.

Also simplify how the predict_satellite_overpass function stores valid overpasses.

  • Overpasses are stored to a dictionary using keys named as <satellite>_<sector>_<time> instead of pass_<N>.

modified: geoips/sector_utils/overpass_predictor.py

Add new kwarg to sector create_test_plot to return figure, axis, and map objects#

Add new return_fig_ax_map kwarg to the create_test_plot function under the sector interface, which returns the fig, ax, and mapobj.

  • This allows other scripts/modules to quickly create an image for a specific sector.

  • Also check if fname is None. If None, create_test_plot does not save the image.

modified: geoips/interfaces/yaml_based/sectors.py

Add support for creating product composites in config_based processing#

From GEOIPS#461: 2024-02-09, add composite support in config_based procflow

Add support for compositing products in the config_based procflow.Procflow finds pre-processed algorithm netcdf files for a sector that exists on disk, reads them into memory, then combines to create a composited product.

Add new xarray data utility that combined multiple xarray datasets of the same dimensions together. This is used for creating a composite using pre-processed algorithms stored to netCDF files, and with the output from an algorithm produced from the current procflow.

Add new argument group - composite_group. Add new command line argument to this group for overriding the composite kwargs for supported sector outputs.

New geoips utility for finding input files for generating a composite. Currently only supports finding netCDF files checked into the product database, but can/will be expanded to support other formats along with no reliance on a database.

modified:   geoips/commandline/args.py
modified:   geoips/plugins/modules/procflows/config_based.py
modified:   geoips/xarray_utils/data.py
new file:   geoips/utils/composite.py

Installation Updates#

Update pyproject.toml dynamic versioning to strict=false#

This allows using version 0.0.0 if no tags are set. If strict=true, unable to pip install package without a tag.

modified: pyproject.toml

Raise error when “interface” not defined on plugin during create_plugin_registries#

Also, raise informative error if a plugin doesn’t exist on base.py. Additionally add indent=4 to make json registries human readable (this is also incoming from github.com)

modified: geoips/create_plugin_registries.py
modified: geoips/interfaces/base.py

Add check_system_requirements function to install mamba package#

When mamba_install passed, install the package specified command line via mamba.

modified: setup/check_system_requirements.sh

Add check_system_requirements functions to check_command and check_environment_variable#

When check_command is passed, check if “command –version” returns 0, if not, exit 1.

When check_environment_variable is passed, check that the env var is defined, if not, exit 1.

modified: setup/check_system_requirements.sh

Add check_system_requirements function to run arbitrary commands#

When “run_command” passed to check system requirements, just run that command, and exit if the command returns non-zero.

modified: setup/check_system_requirements.sh

Update check_system_requirements to allow GEOIPS_MODIFIED_BRANCH#

If GEOIPS_MODIFIED_BRANCH is set, switch to it when cloning a repo. This is not a REQUIRED branch, if it doesn’t exist, just stay where we are (this is for multi-repo issues, where the same branch is on all the related repos)

modified: setup/check_system_requirements.sh

Add -v option to pip install commands in check_system_requirements.sh#

modified: setup/check_system_requirements.sh

Update check_system_requirements to support pulling test data from github#

Allow passing github URL for test data downloads. Do not attempt to decompress github cloned test data.

modified: setup/check_system_requirements.sh
modified: setup/download_test_data.py

Update installation.rst to use standard installation process#

From GEOIPS#446: 2024-01-11, clarify installation and git workflow

For consistency between github.com and internal, ensure installation.rst uses the same process both internal and public. No longer use setup.sh steps, just include the wget, mamba, and pip steps explicitly.

deleted: setup.sh
modified: docs/source/starter/installation.rst

Add create_plugin_registries after base_install.sh#

modified: docs/source/starter/installation.rst
modified: docs/source/starter/mac_installation.rst

Update from Python 3.9 to 3.10 for default install#

modified: docs/source/starter/installation.rst

GitHub Actions Updates#

Add GitHub pages documentation deployment workflow#

From GEOIPS#235: 2024-01-05, deploy ghpages action

Update deploy_pages.sh script to take arguments for GEOIPS_REPO_URL and the github pages repo clone locations (rather than using environment variables).

Add workflow to build html and pdf documentation, then deploy to ghpages. Runs on published release.

Ensure the “ghpages_repo” directory is removed prior to running deploy_pages.sh - it will not run if that directory exists, and it will persist between github actions runs.

.github/workflows/deploy-ghpages-docs.yaml
docs/deploy_pages.sh

Finalize HTML/pdf build for PR validation#

Update to appropriate mamba and pip commands. This workflow only builds the html docs, does not deploy. Need a separate workflow for building and deploying on release.

Additionally include pdf build.

modified: .github/workflows/build-html-docs.yml
new: .github/workflows/build-pdf-docs.yml

Allow calling build_docs.sh without GEOIPS_PACKAGES_DIR set#

Allow passing full path to geoips docs directory, only use $GEOIPS_PACKAGES_DIR if docs directory is not passed in.

modified: docs/build_docs.sh

Documentation Updates#

Add comments regarding flake8 ignores and interface = None#

modified: geoips/interfaces/__init__.py
modified: geoips/plugins/modules/filename_formatters/utils/tc_file_naming.py
modified: geoips/plugins/modules/interpolators/utils/interp_pyresample.py
modified: geoips/plugins/modules/interpolators/utils/interp_scipy.py
modified: geoips/plugins/modules/readers/utils/geostationary_geolocation.py
modified: geoips/plugins/modules/readers/utils/hrit_reader.py
modified: geoips/plugins/modules/readers/utils/remss_reader.py

Ignore */lib/* directories for sphinx api doc#

Currently documentation builds including fortran libraries built with f2py fail - ignore the “lib” directory when building documentation for now, perhaps in the future determine how to build docuemntation for pre-built libraries.

modified: docs/build_docs.sh

Require updating release note directly rather than CHANGELOG.rst#

From GEOIPS#446: 2024-01-11, clarify installation and git workflow

Previously allowed developers to add their changes to CHANGELOG.rst, then move during the PR review process. For simplicity and consistency, just point people to the “update_this_release_note” file to identify the correct release note, and have them update that directly. This involved updating instructions in 3 RST files (CHANGELOG, CHANGELOG_TEMPLATE, and git_workflow).

modified: CHANGELOG.rst
modified: CHANGELOG_TEMPLATE.rst
modified: docs/source/devguide/git_workflow.rst

Update sphinx conf.py to support latex pdf build#

conf_PKG.py required enumitem list depth of 99 to successfully build latex pdf output. Default list depth of 6 is not compatible with sphinx auto API documentation.

modified: docs/source/_templates/conf_PKG.py

Add documentation for reader unit testing#

modified:   docs/source/devguide/unit_tests.rst

Testing Updates#

Fail on realtime test if copy file dir does not exist#

To avoid making it through an entire test if the appropriate directory is not defined for storing test outputs, fail immediately if incorrect directory is specified.

modified: tests/utils/get_realtime_test_args.sh

Single pixel difference update in AHI WV TC GEOTIFF test output#

Dependency update? diff of hexdump of original and updated image show a single bit difference. tifffile updated from 2023.9.26 to 2024.02.12.

bash$ diff currentdump.txt olddump.txt | more
127c127
< 00000820  3e af 7f 67 82 3f d3 38  3e af 7f 67 82 3f 00 00  |>..g.?.8>..g.?..|
---
> 00000820  3e af 7f 67 82 3f d2 38  3e af 7f 67 82 3f 00 00  |>..g.?.8>..g.?..|

modified:   tests/outputs/ahi.tc.WV.geotiff/20200405_000000_SH252020_ahi_himawari-8_WV_100kts_100p00_1p0.tif

Single pixel difference update in ABI Visible global test output#

Must be a dependency update. Literally one pixel. xarray updated from 2023.11.0 to 2023.12.0, could be that.

M tests/outputs/abi.static.Visible.imagery_annotated/20200918.195020.goes-16.abi.Visible.goes_east.41p12.noaa.10p0.png

Update reader unit tests to support multiple tests, and comparing mean#

Return a list of parameters for reader unit testing rather than a single set of parameters. This allows testing multiple datasets if desired.

Additionally, allow including the expected mean from the reader unit test, to confirm the data values are as expected.

modified:   tests/unit_tests_long/plugins/modules/readers/test_readers.py
modified:   geoips/plugins/modules/readers/amsr2_remss_winds_netcdf.py
modified:   geoips/plugins/modules/readers/gmi_hdf5.py
modified:   geoips/plugins/modules/readers/sar_winds_netcdf.py
modified:   geoips/plugins/modules/readers/scat_knmi_winds_netcdf.py
modified:   geoips/plugins/modules/readers/scat_noaa_winds_netcdf.py
modified:   geoips/plugins/modules/readers/smap_remss_winds_netcdf.py

Release Process#

Add release note for v1.12.1#

From GEOIPS#422: 2023-12-13, 1.12.0 release process updates

All updates until the next release (v1.12.1) will be included in this release note.

modified: docs/source/releases/v1.12.1.rst
modified: docs/source/releases/index.rst

Update ‘update_this_release_note’#

From GEOIPS#422: 2023-12-13, 1.12.0 release process updates

All updates until the next release (v1.12.1) will be included in the release note specified in this file.

modified: update_this_release_note
On this page

This Page