# # # Distribution Statement A. Approved for public release. Distribution unlimited.# # ## # # Author:# # # Naval Research Laboratory, Marine Meteorology Division# # ## # # This program is free software: you can redistribute it and/or modify it under# # # the terms of the NRLMMD License included with this program. This program is# # # distributed WITHOUT ANY WARRANTY; without even the implied warranty of# # # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the included license# # # for more details. If you did not receive the license, for more information see:
Version 1.5.2 (2022-09-26)#
v1.5.2: 2022-09-26, testing updates, interp fixes, update xrdict outputs
Bug fixes#
Force cartopy==0.20.3, matplotlib==3.5.3
cartopy 0.20.3 incompatible with matplotlib 3.6.0
cartop 0.21.0 works with 3.6.0, but incompatible with some proj libraries
Remove instances of xarray.ufuncs.logical_and
No longer available at least as of v2022.06 (was available < 0.19.0)
Use numpy.logical_and instead
tests/outputs/smap.unsectored.text_winds/smap-spd_rss_smap_surface_winds_20210926.0000.txt.gz
Add SMAP output file back in (accidentally deleting with previous commit)
geoips/interface_modules/readers/gmi_hdf5.py
Sort original_source_filenames so consistent order in YAML metadata outputs
NetCDF Diff output#
GEOIPS/geoips#95: 2022-09-26, add additional information for netcdf diff output
Testing Improvements#
modified: geoips/compare_outputs.py
When assert_allclose or assert_identical fail, include min/max/mean diff information in log output
assert_allclose output limited.
ASCAT UHR reader and products#
GEOIPS/geoips#92: 2022-09-21, update ASCAT UHR reader and products
Major New Functionality#
modified: geoips/interface_modules/readers/ascat_uhr_netcdf.py
Use _B_ and _C_ in filenames to determine METOP-B vs METOP-C
Use updated filename format for storm name and date MUIFA_20220911_19947_C_D-product.nc
DO NOT subtract 90 from latitude
Use single “sig” variable rather than sig_fore and sig_aft
Use time array, and temporarily use YYYYMMDD from filename for units (bug upstream)
modified: geoips/yaml_configs/product_inputs/ascatuhr.yaml
Add sectored and unsectored text outputs
modified: geoips/yaml_configs/product_inputs/ascat.yaml
Add sectored and unsectored text outputs
UNMODIFIED: geoips/yaml_configs/sectors_dynamic/tc_web_ascatuhr_barbs_template.yaml
new file: geoips/yaml_configs/sectors_dynamic/tc_huge/tc_0p1km_3200x3200.yaml
100m resolution, 3200x3200, useful for zoomed in wind barbs.
new: geoips/yaml_configs/plotting_params/gridlines/tc_0p25degree.yaml
Add quarter degree gridlines - useful for zoomed in windbarbs plots
Error handling in overpass predictor#
GEOIPS/geoips#90: 2022-09-21, expand error handling in overpass predictor
Bug fixes#
geoips/sector_utils/overpass_predictor.py
Add handling for TypeError when calculating the next overpass
Image Comparison Fuzz#
GEOIPS/geoips#84: 2022-09-13, update image comparison fuzz from 1% to 5%
Installation and Testing#
geoips/compare_outputs.py: update image comparison fuzz from 1% to 5%
update xrdict output formats#
GEOIPS/geoips#84: 2022-09-13, update xrdict output formats
Refactor#
geoips/dev/output.py: consolidate xarray_dict based output format families
‘xarray_dict_to_image’ -> ‘xrdict_area_varlist_to_outlist’
‘xarray_dict_data’ -> ‘xrdict_varlist_outfnames_to_outlist’
‘xarray_dict’ -> ‘xrdict_product_outfnames_to_outlist’
xarray_datasets -> xarray_dict
xarray_objs -> xarray_dict
geoips/interface_modules/procflows/single_source.py: update output format if statements for xarray_dict-based
xarray_dict_data -> xrdict_varlist_outfnames_to_outlist
xarray_dict -> xrdict_area_product_outfnames_to_outlist
make product_name_title and mpl_colors_info optional for xrdic_area_product_outfnames_to_outlist family
geoips/interface_modules/output_formats/text_winds.py: Update for xarray_dict-based standards
xarray_dict_data -> xrdict_varlist_outfnames_to_outlist
xarray_objs -> xarray_dict
product_names -> varlist
add “force_alt_varname” to coverage checks#
GEOIPS/geoips#80: 2022-09-12, add “force_alt_varname” to coverage checks
Improvements#
Add “force_alt_varname” option to coverage checks to force using alt_varname_for_coverage
modified: geoips/interface_modules/coverage_checks/center_radius.py
modified: geoips/interface_modules/coverage_checks/center_radius_rgba.py
modified: geoips/interface_modules/coverage_checks/masked_arrays.py
modified: geoips/interface_modules/coverage_checks/numpy_arrays_nan.py
modified: geoips/interface_modules/coverage_checks/rgba.py
resolve missing lat/lon issues in interp#
GEOIPS/geoips#78: 2022-09-10, resolve missing lat/lon issues in interp
Bug fixes#
geoips/interface_modules/interpolation/pyresample_wrappers/interp_gauss.py
geoips/interface_modules/interpolation/pyresample_wrappers/interp_nearest.py
geoips/interface_modules/interpolation/scipy_wrappers/interp_grid.py
If “output_xarray” is None, intiialize to xarray.Dataset()
If ‘latitude’ is not in output_xarray.variables, then calculate and add all geolocation information to Dataset
This allows passing an empty xarray Dataset OR None for output_xarray, and still having proper functionality
allow copying files when checking file list#
GEOIPS/geoips_system_nrl#29: 2022-09-09, allow copying files when checking file list
Installation and Test#
tests/utils/check_output_file_list.sh
If “copy_dir” is specified, copy files to appropriate location
subdirectory based on extension
“noext” subdirectory if no “.” in filename
gunzip and rename files to “.tif” if “.jif.gz” extension
Also include original .gz file separately from unzipped version
tests/utils/get_realtime_test_args.sh
Provide common args for realtime test script setup, so if they need updated, they will be in one place.
bug fixes and updates#
GEOIPS/geoips#75: 2022-09-07, 1.5.2 bug fixes and updates
Installation and Test#
base_install_and_test.sh: Remove seviri, vim8, vim8_plugin, and natural-earth-vector setup comments
Bug fixes#
geoips/interface_modules/procflows/single_source.py
Move “copy_standard_metadata” after interpolation, before calling algorithm
Previously True Color algorithm was failing due to missing “source_name” attribute on xarray Dataset.
geoips/interface_modules/procflows/config_based.py
Skip background data if no coverage (do not fail catastrophically)
geoips/interface_modules/readers/utils/geostationary_geolocation.py
Raise “CoverageError” if there are no good_lines or good_samples
drop_nan, include lat/lon if masked#
GEOIPS/geoips_tempest#5: 2022-09-09, for drop_nan, include lat/lon if masked
Bug fixes#
geoips/interface_modules/interpolation/pyresample_wrapper/interp_gauss.py
If “drop_nan=True” ensure lat/lon masking is included in the overall mask.
v1.5.2.dev2: 2022-09-02, image_utils numpy docstrings, basic CI
add non-member forking info to git-workflow#
NRLMMD-GEOIPS/geoips#17: 2022-08-12, add non-member forking info to git-workflow
Documentation#
docs/git-workflow.rst
Add non-member forking process
Note under branching instructions that branching only applies to members of NRLMMD-GEOIPS, non-members must follow forking instructions
Update image_utils to numpy docstrings#
NRLMMD-GEOIPS/geoips#29: 2022-08-28, Update image_utils to numpy docstrings
Documentation#
Update image_utils directory for proper numpy style docstrings
geoips/image_utils/__init__.py
geoips/image_utils/colormap_utils.py
geoips/image_utils/maps.py
geoips/image_utils/mpl_utils.py
geoips/compare_outputs.py Update imagemagick compare metric from rmse to ae + fuzz 1%
Add Dockerfile and Basic CI#
NRLMMD-GEOIPS/geoips#34 - 2022-08-12 - Add Dockerfile and Basic CI
Installation and Test#
.dockerignore
Add .dockerignore
.github/workflows/build-and-test-in-docker.yaml
Add a basic workflow that builds a docker images and pushes it to the GitHub package registry
Dockerfile
Add a dockerfile that builds an image containing a working version of GeoIPS
base_install_and_test.sh
Remove conda_link step
Directly source bashrc
geoips/filenames/base_paths.py
Add BASE_PATH to PATHS{} and collect it using pathjoin(dirname(__file__), ‘..’)
Remove PATHS[‘GEOIPS’]
Use BASE_PATH to find TC_TEMPLATE path
geoips/image_utils/maps.py
Add some debug statements
geoips/interface_modules/title_formats/__init__.py
Add an __init__.py here so this can be imported correctly
geoips/interface_modules/user_colormaps/tpw/tpw_cimss.py
Use BASE_PATH rather than GEOIPS
geoips/interface_modules/user_colormaps/tpw/tpw_purple.py
Use BASE_PATH rather than GEOIPS
geoips/interface_modules/user_colormaps/tpw/tpw_pwat.py
Use BASE_PATH rather than GEOIPS
geoips/utils/__init__.py
Add an __init__.py here so this can be imported correctly
setup.py
Add use of package_data for yaml_configs and image_utils/ascii_palettes
Allow pip install of pyshp, shapely, and cartopy
Move install of pyshp from test_outputs to main install_requires
setup.sh
Add creation of $GEOIPS_DEPENDENCIES_DIR/bin at top of script
Remove conda_link action
Directly call conda init rather than $BASECONDAPATH/conda init (assumes conda is in $PATH) from sourcing either ~/.bashrc or setup/config_geoips
Use GEOIPS_TESTDATA_DIR rather than $GEOIPS_PACKAGES_DIR/geoips/tests/data/
tests/scripts/abi.static.Infrared.imagery_annotated.sh
Replace all references to GEOIPS/tests/data/ with GEOIPS_TESTDATA_DIR
tests/scripts/abi.static.Visible.imagery_annotated.sh
Replace all references to GEOIPS/tests/data/ with GEOIPS_TESTDATA_DIR
tests/scripts/amsr2_ocean.tc.windspeed.imagery_clean.sh
Replace all references to GEOIPS/tests/data/ with GEOIPS_TESTDATA_DIR
tests/scripts/documentation_imagery.sh
Replace all references to GEOIPS/tests/data/ with GEOIPS_TESTDATA_DIR
tests/yaml_configs/abi_test_low_memory.yaml
Replace all references to GEOIPS/tests/data/ with GEOIPS_TESTDATA_DIR
tests/yaml_configs/abi_test.yaml
Replace all references to GEOIPS/tests/data/ with GEOIPS_TESTDATA_DIR
support data_fusion#
GEOIPS/pyrocb#7: 2022-08-30, support data_fusion
Refactor#
geoips/dev/product.py: Replace “base_product_name” with “product_template” in get_product
product_name: Actual name of current product
product_template: YAML file to use as the base for the current product
product_category: A specification that allows grouping “similar” products (ie, 37H, 34H, 36H all in the 37H “product_category”)
Major New Functionality#
geoips/yaml_configs/product_params/alg.yaml: Template containing:
product_type=’alg’
geoips/yaml_configs/product_params/interp.yaml: Template containing:
product_type=’interp’
interp_func default: pyresample_wrappers.interp_nearest
interp_args default: {}
geoips/yaml_configs/product_params/unmodified.yaml: Template containing:
product_type=’unmodified’
tests/sectors/tc_bdecks/bep072022.dat
EP07Frank sample bdeck file
Improvements#
geoips/interface_modules/procflows/single_source.py: Support data_fusion functionality
def pad_area_definition
Allow passing “force_pad” for non-TC sectors
Allow passing x_scale_factor and y_scale_factor for different scaling factors
def plot_data: Support xarray_dict output format type.
def get_alg_xarray: Support data_fusion processing
If variable_names is passed, use it (impacts reader_defined and self_registered products)
set “alg_func_type” to None if no algorithm defined.
Default “interp_xarray” to xarray.Dataset() rather than None
Reassign interp_func within the interpolation loop, to ensure it is using the current sect_xarray source for definiing the appropriate interpolation routine.
If “time” is contained in sect_xarray dims, interpolate each slice of the array separately
Copy standard metadata to “interp_xarray” before returning, using “force=False”
geoips/dev/utils.py
Add “force” option to copy_standard_metadata - allow NOT replacing existing fields.
This will not impact existing functionality - default is force=True, and when force=True the original if statement will always be used.
geoips/geoips_utils.py
Pass “force” option directly through to dev.utils.copy_standard_metadata
allow drop_nan option for interp_gauss#
GEOIPS/geoips#65: 2022-08-20, allow drop_nan option for interp_gauss
Improvements#
geoips/interface_modules/pyresample_wrappers/interp_gauss.py
If drop_nan=True passed, remove all values from lat/lon/data arrays where any array contains numpy.nan
If drop_nan=False, operation remains unchanged (backwards compatible)
allow specifying base_product_name within product_inputs#
GEOIPS/geoips#63: 2022-08-20, allow specifying base_product_name within product_inputs
Major New Functionality#
geoips/dev/product.py: Use base_product_name if specified in product_inputs dict for retrieving product params
- This allows specifying new product names directly within the product_inputs YAMLs, rather than
requiring a separate YAML file for every product name.
- Useful for classes of products that have all the same parameters, except potentially a different filename /
required variable (ie, 34GHz, 35GHz, 36GHz and 37GHz products can now reuse the same 37H product, but have unique filenames)
generalized NOAA AWS download script#
GEOIPS/geoips#65: 2022-08-20, generalized NOAA AWS download script
Major New Functionality#
setup.sh: Update setup_abi_test_data to use download_noaa_aws.sh script.
tests/download_noaa_aws.sh: Allows downloading specific satellite, YYYYMMDD.HHMN of data from NOAA AWS.
tests/scripts/abi.static.Infrared.imagery_annotated.sh: Update GOES16 path to use $GEOIPS_TESTDATA_DIR
tests/scripts/abi.static.Visible.imagery_annotated.sh: Update GOES16 path to use $GEOIPS_TESTDATA_DIR
tests/scripts/documentation_imagery.sh: Update GOES16 path to use $GEOIPS_TESTDATA_DIR
tests/yaml_configs/abi_test.yaml: Update GOES16 path to use $GEOIPS_TESTDATA_DIR
tests/yaml_configs/abi_test_low_memory.yaml: Update GOES16 path to use $GEOIPS_TESTDATA_DIR
new plugin process#
GEOIPS/geoips#62: 2022-08-21, new plugin process
Documentation#
docs/setup-new-plugin.rst: Process for setting up a new test date repo, and plugin repo.
docs/version_control_templates.rst: Remove rst - these are now direct GitHub templates.
docs/geoips_index.rst: Remove version_control_templates.rst, add git-workflow and setup-new-plugin
support CARTOPY_DATA_DIR#
GEOIPS/geoips#57: 2022-08-17, support CARTOPY_DATA_DIR
Installation and Test#
setup.sh: Link cartopy data into $CARTOPY_DATA_DIR vs ~/.local/share/cartopy
config_geoips: Set CARTOPY_DATA_DIR to $GEOIPS_DEPENDENCIES_DIR/CARTOPY_DATA_DIR
setup.py: Add shapely, cartopy, and pyshp requirements. cartopy error if not specified.
allow less strict image comparisons#
GEOIPS/geoips#59: 2022-08-18, allow less strict image comparisons
Improvements#
geoips/compare_outputs.py
Add “fuzz” argument to “images_match” function - default to 1%
Add “fuzz” argument to imagemagick compare call
Change metric from rmse to ae (RMSE did not appear to care about the fuzz factor)
Re-run compare with RMSE and no fuzz factor if test passes (to track differences)
# v1.5.2.dev1: 2022-08-16, improve install, test, and git workflow
add test datasets to .gitignore#
NRLMMD-GEOIPS/geoips#31 - add test datasets to .gitignore
Improvements#
geoips/.gitignore
Add tests/data/** to .gitignore so they no longer appear in git status
add low_bandwidth option#
NRLMMD-GEOIPS/geoips#25 - add low_bandwidth option
Installation and Test#
README.md - Pass low_memory and low_bandwidth options to base_install_and_test.sh
setup.sh - Support low_bandwidth option for setup_abi_test_data (only download B14), and install (pip install minimum packages for tests)
base_install_and_test.sh - Pass “low_bandwidth” option through to setup.sh
NRLMMD-GEOIPS/geoips#27#
Installation and Test#
Add “check_system_requirements.sh” script to ensure system requirements are installed
git lfs
imagemagick
wget
recent git version
Update Git Workflow#
NRLMMD-GEOIPS/geoips#17 - Update Git Workflow
Documentation Updates#
docs/git-workflow.rst
Remove manual labeling on Issues
Remove manual labeling on PRs
Remove manual branching command line
Note that branches MUST be created via the Issue->Development->Create Branch option
Remove manual status updates on Project (should be automated via PRs linked to Issue)
Remove rclone.conf link#
NRLMMD-GEOIPS/geoips#22 - Remove rclone.conf link to ~/.config/rclone
Installation and Test#
setup.sh#
- Update setup_rclone command to remove the link from $GEOIPS_PACKAGES_DIR/geoips/setup/rclone_setup/rclone.conf
to ~/.config/rclone/rclone.conf
- Update setup_abi_test_data command to use explicit
–config $GEOIPS_PACKAGES_DIR/geoips/setup/rclone_setup/rclone.conf argument rather than relying on default ~/.config/rclone/rclone.conf configuration
Add low memory options for base install tests#
NRLMMD-GEOIPS/geoips#15 - Add low memory options for base install tests
Test Repo Updates#
abi.config_based_output_low_memory.sh
abi.static.Infrared.imagery_annotated png output
abi.tc.Infrared.imagery_annotated png and YAML metadata output
abi.tc.IR-BD.imagery_annotated png and YAML metadata output
abi.static.Infrared.imagery_annotated.sh
abi.static.Infrared.imagery_annotated png output
amsr2.config_based_overlay_output_low_memory.sh
amsr2.global_overlay.37pct.imagery_annotated_over_Infrared-Gray png and YAML metadata output
amsr2.global_overlay.89pct.imagery_annotated_over_Infrared-Gray png and YAML metadata output
amsr2.tc_overlay.37pct.imagery_annotated_over_Infrared-Gray png and YAML metadata output
amsr2.tc_overlay.89pct.imagery_annotated_over_Infrared-Gray png and YAML metadata output
UPDATE outputs amsr2.config_based_overlay_output.sh (outputs were not previously included)
amsr2.global_overlay.37pct.imagery_annotated_over_Visible png and YAML metadata output
amsr2.global_overlay.89pct.imagery_annotated_over_Visible png and YAML metadata output
amsr2.tc_overlay.37pct.imagery_annotated_over_Visible png and YAML metadata output
amsr2.tc_overlay.89pct.imagery_annotated_over_Visible png and YAML metadata output
Installation and Test#
base_install_and_test.sh
- Add “low_memory” option that allows testing Infrared-only ABI rather than Visible.
~4GB vs ~12GB memory requirement.
Bug fixes#
amsr2.config_based_overlay_output.sh
Un-indent “backgrond_products” so background imagery is included in outputs
Add outputs to comparison directories
Add AMSR2 test data and test scripts#
NRLMMD-GEOIPS/geoips_tutorial#3 - Add AMSR2 test data and test scripts to base install and test
Installation and Test#
README.md
Add git lfs install to setup, to ensure Large File Storage tracked data files are cloned properly
base_install_and_test.sh
Add clone of test_data_amsr2
Add AMSR2 test: $GEOIPS_PACKAGES_DIR/geoips/tests/scripts/amsr2.config_based_overlay_output.sh
setup.py
Add scikit-image to “coverage_checks” section of install_requires
config_geoips
Add git lfs install, for redundancy
Add GEOIPS_TESTDATA_DIR environment variable, to allow non-GEOIPS BASEDIR test data locations.
AMSR2 Test Scripts
Add AMSR2 config based test script: tests/scripts/amsr2.config_based_overlay_output.sh
Add AMSR2 YAML output config: tests/yaml_configs/amsr2_test.yaml
89pct and 37pct output products
TC-centric sector
Global sector
Visible AHI background imagery
Streamline installation process, support Mac installation#
NRLMMD-GEOIPS/geoips#6,8,9,11 - Streamline installation process, support Mac installation
Installation and Test#
base_install_and_test.sh#
Exit immediately if GEOIPS BASEDIR or GEOIPS_REPO_URL are not defined
Comment out several sections of installation, to reduce time and disk space
natural-earth-vector data download (will rely on latest shapefiles during cartopy processing)
natural-earth-vector linking to ~/.local/share/cartopy
Will NOT reinstate this step - cartopy supports CARTOPY_DATA_DIR as of 6 August 2021
vim8 installation (only for use of vim8 plugins to help with following style guides)
vim8 plugin installation
seviri setup
Remove BASECONDAPATH from conda cartopy installation (conda will be in PATH)
setup.sh#
- To support Mac installations, use “uname -m” when determining filenames for
rclone and miniconda3 installation
Rather than sourcing .bashrc to get the conda environment set up, source geoips_conda_init_setup.
Update default branches from dev to main
geoips_conda_init_setup#
- To support Mac installations, use $(conda shell.bash activate geoips_conda) when activating
conda vs “conda geoips_conda activate”
- Allow use of GeoIPS-specific conda installation along-side user/system level installation where
the user/system level installation may be initialized in .bash_profile. Uses GeoIPS-specific installation by default, if it is found.
color_prompt#
Add “$CONDA_PROMPT_MODIFIER” to $PS1
repo_clone_update_install.sh#
- If GEOIPS_TESTDATA_DIR, GEOIPS_PACKAGES_DIR, or GEOIPS_DEPENDENCIES_DIR are set, use those,
otherwise default to placing under $GEOIPS BASEDIR
Update default branch from dev to main
README.md#
Update github.com GEOIPS_ACTIVE_BRANCH from dev to main