| # # # Distribution Statement A. Approved for public release. Distribution unlimited. | # # # | # # # Author: | # # # Naval Research Laboratory, Marine Meteorology Division | # # # | # # # This program is free software: you can redistribute it and/or modify it under | # # # the terms of the NRLMMD License included with this program. This program is | # # # distributed WITHOUT ANY WARRANTY; without even the implied warranty of | # # # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the included license | # # # for more details. If you did not receive the license, for more information see: | # # # https://github.com/U-S-NRL-Marine-Meteorology-Division/ Version 1.5.2 (2022-09-26) ************************** v1.5.2: 2022-09-26, testing updates, interp fixes, update xrdict outputs Bug fixes ========= * Force cartopy==0.20.3, matplotlib==3.5.3 * cartopy 0.20.3 incompatible with matplotlib 3.6.0 * cartop 0.21.0 works with 3.6.0, but incompatible with some proj libraries * Remove instances of xarray.ufuncs.logical_and * No longer available at least as of v2022.06 (was available < 0.19.0) * Use numpy.logical_and instead * **tests/outputs/smap.unsectored.text_winds/smap-spd_rss_smap_surface_winds_20210926.0000.txt.gz** * Add SMAP output file back in (accidentally deleting with previous commit) * **geoips/interface_modules/readers/gmi_hdf5.py** * Sort original_source_filenames so consistent order in YAML metadata outputs NetCDF Diff output ================== GEOIPS/geoips#95: 2022-09-26, add additional information for netcdf diff output Testing Improvements -------------------- * modified: **geoips/compare_outputs.py** * When assert_allclose or assert_identical fail, include min/max/mean diff information in log output * assert_allclose output limited. ASCAT UHR reader and products ============================= GEOIPS/geoips#92: 2022-09-21, update ASCAT UHR reader and products Major New Functionality ----------------------- * modified: **geoips/interface_modules/readers/ascat_uhr_netcdf.py** * Use _B_ and _C_ in filenames to determine METOP-B vs METOP-C * Use updated filename format for storm name and date MUIFA_20220911_19947_C_D-product.nc * DO NOT subtract 90 from latitude * Use single "sig" variable rather than sig_fore and sig_aft * Use time array, and temporarily use YYYYMMDD from filename for units (bug upstream) * modified: **geoips/yaml_configs/product_inputs/ascatuhr.yaml** * Add sectored and unsectored text outputs * modified: **geoips/yaml_configs/product_inputs/ascat.yaml** * Add sectored and unsectored text outputs * UNMODIFIED: **geoips/yaml_configs/sectors_dynamic/tc_web_ascatuhr_barbs_template.yaml** * new file: **geoips/yaml_configs/sectors_dynamic/tc_huge/tc_0p1km_3200x3200.yaml** * 100m resolution, 3200x3200, useful for zoomed in wind barbs. * new: **geoips/yaml_configs/plotting_params/gridlines/tc_0p25degree.yaml** * Add quarter degree gridlines - useful for zoomed in windbarbs plots Error handling in overpass predictor ==================================== GEOIPS/geoips#90: 2022-09-21, expand error handling in overpass predictor Bug fixes --------- * **geoips/sector_utils/overpass_predictor.py** * Add handling for TypeError when calculating the next overpass Image Comparison Fuzz ===================== GEOIPS/geoips#84: 2022-09-13, update image comparison fuzz from 1% to 5% Installation and Testing ------------------------ * **geoips/compare_outputs.py**: update image comparison fuzz from 1% to 5% update xrdict output formats ============================ GEOIPS/geoips#84: 2022-09-13, update xrdict output formats Refactor -------- * **geoips/dev/output.py**: consolidate xarray_dict based output format families * 'xarray_dict_to_image' -> 'xrdict_area_varlist_to_outlist' * 'xarray_dict_data' -> 'xrdict_varlist_outfnames_to_outlist' * 'xarray_dict' -> 'xrdict_product_outfnames_to_outlist' * xarray_datasets -> xarray_dict * xarray_objs -> xarray_dict * **geoips/interface_modules/procflows/single_source.py**: update output format if statements for xarray_dict-based * xarray_dict_data -> xrdict_varlist_outfnames_to_outlist * xarray_dict -> xrdict_area_product_outfnames_to_outlist * make product_name_title and mpl_colors_info optional for xrdic_area_product_outfnames_to_outlist family * **geoips/interface_modules/output_formats/text_winds.py**: Update for xarray_dict-based standards * xarray_dict_data -> xrdict_varlist_outfnames_to_outlist * xarray_objs -> xarray_dict * product_names -> varlist add "force_alt_varname" to coverage checks ========================================== GEOIPS/geoips#80: 2022-09-12, add "force_alt_varname" to coverage checks Improvements ------------ * Add "force_alt_varname" option to coverage checks to force using alt_varname_for_coverage * modified: **geoips/interface_modules/coverage_checks/center_radius.py** * modified: **geoips/interface_modules/coverage_checks/center_radius_rgba.py** * modified: **geoips/interface_modules/coverage_checks/masked_arrays.py** * modified: **geoips/interface_modules/coverage_checks/numpy_arrays_nan.py** * modified: **geoips/interface_modules/coverage_checks/rgba.py** add shared uncompress_test_data script ====================================== GEOIPS/geoips#82: 2022-09-13, add shared uncompress_test_data script Major New Functionality ----------------------- * **tests/utils/uncompress_test_data.sh** * Utility that will decompress gz, bz2, or tgz data files within the passed directory * This can be used by individual repos to decompress their test and output data files prior to processing * This script is not called automatically - must be called by the individual repo's uncompress_test_data.sh resolve missing lat/lon issues in interp ======================================== GEOIPS/geoips#78: 2022-09-10, resolve missing lat/lon issues in interp Bug fixes --------- * **geoips/interface_modules/interpolation/pyresample_wrappers/interp_gauss.py** * **geoips/interface_modules/interpolation/pyresample_wrappers/interp_nearest.py** * **geoips/interface_modules/interpolation/scipy_wrappers/interp_grid.py** * If "output_xarray" is None, intiialize to xarray.Dataset() * If 'latitude' is not in output_xarray.variables, then calculate and add all geolocation information to Dataset * This allows passing an empty xarray Dataset OR None for output_xarray, and still having proper functionality allow copying files when checking file list =========================================== GEOIPS/geoips_system_nrl#29: 2022-09-09, allow copying files when checking file list Installation and Test --------------------- * **tests/utils/check_output_file_list.sh** * If "copy_dir" is specified, copy files to appropriate location * subdirectory based on extension * "noext" subdirectory if no "." in filename * gunzip and rename files to ".tif" if ".jif.gz" extension * Also include original .gz file separately from unzipped version * **tests/utils/get_realtime_test_args.sh** * Provide common args for realtime test script setup, so if they need updated, they will be in one place. bug fixes and updates ===================== GEOIPS/geoips#75: 2022-09-07, 1.5.2 bug fixes and updates Installation and Test --------------------- * **base_install_and_test.sh**: Remove seviri, vim8, vim8_plugin, and natural-earth-vector setup comments Bug fixes --------- * **geoips/interface_modules/procflows/single_source.py** * Move "copy_standard_metadata" after interpolation, before calling algorithm * Previously True Color algorithm was failing due to missing "source_name" attribute on xarray Dataset. * **geoips/interface_modules/procflows/config_based.py** * Skip background data if no coverage (do not fail catastrophically) * **geoips/interface_modules/readers/utils/geostationary_geolocation.py** * Raise "CoverageError" if there are no good_lines or good_samples drop_nan, include lat/lon if masked =================================== GEOIPS/geoips_tempest#5: 2022-09-09, for drop_nan, include lat/lon if masked Bug fixes --------- * **geoips/interface_modules/interpolation/pyresample_wrapper/interp_gauss.py** * If "drop_nan=True" ensure lat/lon masking is included in the overall mask. * v1.5.2.dev2: 2022-09-02, image_utils numpy docstrings, basic CI add non-member forking info to git-workflow =========================================== NRLMMD-GEOIPS/geoips#17: 2022-08-12, add non-member forking info to git-workflow Documentation ------------- * **docs/git-workflow.rst** * Add non-member forking process * Note under branching instructions that branching only applies to members of NRLMMD-GEOIPS, non-members must follow forking instructions Update image_utils to numpy docstrings ====================================== NRLMMD-GEOIPS/geoips#29: 2022-08-28, Update image_utils to numpy docstrings Documentation ------------- * Update image_utils directory for proper numpy style docstrings * **geoips/image_utils/__init__.py** * **geoips/image_utils/colormap_utils.py** * **geoips/image_utils/maps.py** * **geoips/image_utils/mpl_utils.py** * **geoips/compare_outputs.py** Update imagemagick compare metric from rmse to ae + fuzz 1% Add Dockerfile and Basic CI =========================== NRLMMD-GEOIPS/geoips#34 - 2022-08-12 - Add Dockerfile and Basic CI Installation and Test --------------------- * **.dockerignore** * Add .dockerignore * **.github/workflows/build-and-test-in-docker.yaml** * Add a basic workflow that builds a docker images and pushes it to the GitHub package registry * **Dockerfile** * Add a dockerfile that builds an image containing a working version of GeoIPS * **base_install_and_test.sh** * Remove conda_link step * Directly source bashrc * **geoips/filenames/base_paths.py** * Add `BASE_PATH` to `PATHS{}` and collect it using `pathjoin(dirname(__file__), '..')` * Remove `PATHS['GEOIPS']` * Use `BASE_PATH` to find `TC_TEMPLATE` path * **geoips/image_utils/maps.py** * Add some debug statements * **geoips/interface_modules/title_formats/__init__.py** * Add an __init__.py here so this can be imported correctly * **geoips/interface_modules/user_colormaps/tpw/tpw_cimss.py** * Use `BASE_PATH` rather than `GEOIPS` * **geoips/interface_modules/user_colormaps/tpw/tpw_purple.py** * Use `BASE_PATH` rather than `GEOIPS` * **geoips/interface_modules/user_colormaps/tpw/tpw_pwat.py** * Use `BASE_PATH` rather than `GEOIPS` * **geoips/utils/__init__.py** * Add an __init__.py here so this can be imported correctly * **setup.py** * Add use of `package_data` for yaml_configs and image_utils/ascii_palettes * Allow pip install of pyshp, shapely, and cartopy * Move install of pyshp from `test_outputs` to main `install_requires` * **setup.sh** * Add creation of `$GEOIPS_DEPENDENCIES_DIR/bin` at top of script * Remove `conda_link` action * Directly call `conda init` rather than `$BASECONDAPATH/conda init` (assumes conda is in $PATH) from sourcing either ~/.bashrc or setup/config_geoips * Use `GEOIPS_TESTDATA_DIR` rather than `$GEOIPS_PACKAGES_DIR/geoips/tests/data/` * **tests/scripts/abi.static.Infrared.imagery_annotated.sh** * Replace all references to `GEOIPS/tests/data/` with `GEOIPS_TESTDATA_DIR` * **tests/scripts/abi.static.Visible.imagery_annotated.sh** * Replace all references to `GEOIPS/tests/data/` with `GEOIPS_TESTDATA_DIR` * **tests/scripts/amsr2_ocean.tc.windspeed.imagery_clean.sh** * Replace all references to `GEOIPS/tests/data/` with `GEOIPS_TESTDATA_DIR` * **tests/scripts/documentation_imagery.sh** * Replace all references to `GEOIPS/tests/data/` with `GEOIPS_TESTDATA_DIR` * **tests/yaml_configs/abi_test_low_memory.yaml** * Replace all references to `GEOIPS/tests/data/` with `GEOIPS_TESTDATA_DIR` * **tests/yaml_configs/abi_test.yaml** * Replace all references to `GEOIPS/tests/data/` with `GEOIPS_TESTDATA_DIR` support data_fusion =================== GEOIPS/pyrocb#7: 2022-08-30, support data_fusion Refactor -------- * **geoips/dev/product.py**: Replace "base_product_name" with "product_template" in get_product * product_name: Actual name of current product * product_template: YAML file to use as the base for the current product * product_category: A specification that allows grouping "similar" products (ie, 37H, 34H, 36H all in the 37H "product_category") Major New Functionality ----------------------- * **geoips/yaml_configs/product_params/alg.yaml**: Template containing: * product_type='alg' * **geoips/yaml_configs/product_params/interp.yaml**: Template containing: * product_type='interp' * interp_func default: pyresample_wrappers.interp_nearest * interp_args default: {} * **geoips/yaml_configs/product_params/unmodified.yaml**: Template containing: * product_type='unmodified' * **tests/sectors/tc_bdecks/bep072022.dat** * EP07Frank sample bdeck file Improvements ------------ * **geoips/interface_modules/procflows/single_source.py**: Support data_fusion functionality * **def pad_area_definition** * Allow passing "force_pad" for non-TC sectors * Allow passing x_scale_factor and y_scale_factor for different scaling factors * **def plot_data**: Support xarray_dict output format type. * **def get_alg_xarray**: Support data_fusion processing * If variable_names is passed, use it (impacts reader_defined and self_registered products) * set "alg_func_type" to None if no algorithm defined. * Default "interp_xarray" to xarray.Dataset() rather than None * Reassign interp_func within the interpolation loop, to ensure it is using the current sect_xarray source for definiing the appropriate interpolation routine. * If "time" is contained in sect_xarray dims, interpolate each slice of the array separately * Copy standard metadata to "interp_xarray" before returning, using "force=False" * **geoips/dev/utils.py** * Add "force" option to copy_standard_metadata - allow NOT replacing existing fields. * This will not impact existing functionality - default is force=True, and when force=True the original if statement will always be used. * **geoips/geoips_utils.py** * Pass "force" option directly through to dev.utils.copy_standard_metadata allow drop_nan option for interp_gauss ====================================== GEOIPS/geoips#65: 2022-08-20, allow drop_nan option for interp_gauss Improvements ------------ * **geoips/interface_modules/pyresample_wrappers/interp_gauss.py** * If drop_nan=True passed, remove all values from lat/lon/data arrays where any array contains numpy.nan * If drop_nan=False, operation remains unchanged (backwards compatible) allow specifying base_product_name within product_inputs ======================================================== GEOIPS/geoips#63: 2022-08-20, allow specifying base_product_name within product_inputs Major New Functionality ----------------------- * **geoips/dev/product.py**: Use base_product_name if specified in product_inputs dict for retrieving product params * This allows specifying new product names directly within the product_inputs YAMLs, rather than requiring a separate YAML file for every product name. * Useful for classes of products that have all the same parameters, except potentially a different filename / required variable (ie, 34GHz, 35GHz, 36GHz and 37GHz products can now reuse the same 37H product, but have unique filenames) generalized NOAA AWS download script ==================================== GEOIPS/geoips#65: 2022-08-20, generalized NOAA AWS download script Major New Functionality ----------------------- * **setup.sh**: Update setup_abi_test_data to use download_noaa_aws.sh script. * **tests/download_noaa_aws.sh**: Allows downloading specific satellite, YYYYMMDD.HHMN of data from NOAA AWS. * **tests/scripts/abi.static.Infrared.imagery_annotated.sh**: Update GOES16 path to use $GEOIPS_TESTDATA_DIR * **tests/scripts/abi.static.Visible.imagery_annotated.sh**: Update GOES16 path to use $GEOIPS_TESTDATA_DIR * **tests/scripts/documentation_imagery.sh**: Update GOES16 path to use $GEOIPS_TESTDATA_DIR * **tests/yaml_configs/abi_test.yaml**: Update GOES16 path to use $GEOIPS_TESTDATA_DIR * **tests/yaml_configs/abi_test_low_memory.yaml**: Update GOES16 path to use $GEOIPS_TESTDATA_DIR new plugin process ================== GEOIPS/geoips#62: 2022-08-21, new plugin process Documentation ------------- * **docs/setup-new-plugin.rst**: Process for setting up a new test date repo, and plugin repo. * **docs/version_control_templates.rst**: Remove rst - these are now direct GitHub templates. * **docs/geoips_index.rst**: Remove version_control_templates.rst, add git-workflow and setup-new-plugin support CARTOPY_DATA_DIR ======================== GEOIPS/geoips#57: 2022-08-17, support CARTOPY_DATA_DIR Installation and Test --------------------- * **setup.sh**: Link cartopy data into $CARTOPY_DATA_DIR vs ~/.local/share/cartopy * **config_geoips**: Set CARTOPY_DATA_DIR to $GEOIPS_DEPENDENCIES_DIR/CARTOPY_DATA_DIR * **setup.py**: Add shapely, cartopy, and pyshp requirements. cartopy error if not specified. allow less strict image comparisons =================================== GEOIPS/geoips#59: 2022-08-18, allow less strict image comparisons Improvements ------------ * **geoips/compare_outputs.py** * Add "fuzz" argument to "images_match" function - default to 1% * Add "fuzz" argument to imagemagick compare call * Change metric from rmse to ae (RMSE did not appear to care about the fuzz factor) * Re-run compare with RMSE and no fuzz factor if test passes (to track differences) # v1.5.2.dev1: 2022-08-16, improve install, test, and git workflow add test datasets to .gitignore =============================== NRLMMD-GEOIPS/geoips#31 - add test datasets to .gitignore Improvements ------------ * **geoips/.gitignore** * Add tests/data/** to .gitignore so they no longer appear in git status add low_bandwidth option ======================== NRLMMD-GEOIPS/geoips#25 - add low_bandwidth option Installation and Test --------------------- * **README.md** - Pass low_memory and low_bandwidth options to base_install_and_test.sh * **setup.sh** - Support low_bandwidth option for setup_abi_test_data (only download B14), and install (pip install minimum packages for tests) * **base_install_and_test.sh** - Pass "low_bandwidth" option through to setup.sh NRLMMD-GEOIPS/geoips#27 ======================= Installation and Test --------------------- * Add "check_system_requirements.sh" script to ensure system requirements are installed * git lfs * imagemagick * wget * recent git version Update Git Workflow =================== NRLMMD-GEOIPS/geoips#17 - Update Git Workflow Documentation Updates --------------------- * **docs/git-workflow.rst** * Remove manual labeling on Issues * Remove manual labeling on PRs * Remove manual branching command line * Note that branches MUST be created via the Issue->Development->Create Branch option * Remove manual status updates on Project (should be automated via PRs linked to Issue) Remove rclone.conf link ======================= NRLMMD-GEOIPS/geoips#22 - Remove rclone.conf link to ~/.config/rclone Installation and Test --------------------- setup.sh ^^^^^^^^ * Update setup_rclone command to remove the link from $GEOIPS_PACKAGES_DIR/geoips/setup/rclone_setup/rclone.conf to ~/.config/rclone/rclone.conf * Update setup_abi_test_data command to use explicit --config $GEOIPS_PACKAGES_DIR/geoips/setup/rclone_setup/rclone.conf argument rather than relying on default ~/.config/rclone/rclone.conf configuration Add low memory options for base install tests ============================================= NRLMMD-GEOIPS/geoips#15 - Add low memory options for base install tests Test Repo Updates ----------------- * abi.config_based_output_low_memory.sh * abi.static.Infrared.imagery_annotated png output * abi.tc.Infrared.imagery_annotated png and YAML metadata output * abi.tc.IR-BD.imagery_annotated png and YAML metadata output * abi.static.Infrared.imagery_annotated.sh * abi.static.Infrared.imagery_annotated png output * amsr2.config_based_overlay_output_low_memory.sh * amsr2.global_overlay.37pct.imagery_annotated_over_Infrared-Gray png and YAML metadata output * amsr2.global_overlay.89pct.imagery_annotated_over_Infrared-Gray png and YAML metadata output * amsr2.tc_overlay.37pct.imagery_annotated_over_Infrared-Gray png and YAML metadata output * amsr2.tc_overlay.89pct.imagery_annotated_over_Infrared-Gray png and YAML metadata output * UPDATE outputs amsr2.config_based_overlay_output.sh (outputs were not previously included) * amsr2.global_overlay.37pct.imagery_annotated_over_Visible png and YAML metadata output * amsr2.global_overlay.89pct.imagery_annotated_over_Visible png and YAML metadata output * amsr2.tc_overlay.37pct.imagery_annotated_over_Visible png and YAML metadata output * amsr2.tc_overlay.89pct.imagery_annotated_over_Visible png and YAML metadata output Installation and Test --------------------- * **base_install_and_test.sh** * Add "low_memory" option that allows testing Infrared-only ABI rather than Visible. ~4GB vs ~12GB memory requirement. Bug fixes --------- * **amsr2.config_based_overlay_output.sh** * Un-indent "backgrond_products" so background imagery is included in outputs * Add outputs to comparison directories Add AMSR2 test data and test scripts ==================================== NRLMMD-GEOIPS/geoips_tutorial#3 - Add AMSR2 test data and test scripts to base install and test Installation and Test --------------------- * **README.md** * Add git lfs install to setup, to ensure Large File Storage tracked data files are cloned properly * **base_install_and_test.sh** * Add clone of test_data_amsr2 * Add AMSR2 test: $GEOIPS_PACKAGES_DIR/geoips/tests/scripts/amsr2.config_based_overlay_output.sh * **setup.py** * Add scikit-image to "coverage_checks" section of install_requires * **config_geoips** * Add git lfs install, for redundancy * Add GEOIPS_TESTDATA_DIR environment variable, to allow non-GEOIPS BASEDIR test data locations. * **AMSR2 Test Scripts** * Add AMSR2 config based test script: tests/scripts/amsr2.config_based_overlay_output.sh * Add AMSR2 YAML output config: tests/yaml_configs/amsr2_test.yaml * 89pct and 37pct output products * TC-centric sector * Global sector * Visible AHI background imagery Streamline installation process, support Mac installation ========================================================= NRLMMD-GEOIPS/geoips#6,8,9,11 - Streamline installation process, support Mac installation Installation and Test --------------------- base_install_and_test.sh ^^^^^^^^^^^^^^^^^^^^^^^^ * Exit immediately if GEOIPS BASEDIR or GEOIPS_REPO_URL are not defined * Comment out several sections of installation, to reduce time and disk space * natural-earth-vector data download (will rely on latest shapefiles during cartopy processing) * natural-earth-vector linking to ~/.local/share/cartopy * Will NOT reinstate this step - cartopy supports CARTOPY_DATA_DIR as of 6 August 2021 * vim8 installation (only for use of vim8 plugins to help with following style guides) * vim8 plugin installation * seviri setup * Remove BASECONDAPATH from conda cartopy installation (conda will be in PATH) setup.sh ^^^^^^^^ * To support Mac installations, use "uname -m" when determining filenames for rclone and miniconda3 installation * Rather than sourcing `.bashrc` to get the conda environment set up, source `geoips_conda_init_setup`. * Update default branches from dev to main geoips_conda_init_setup ^^^^^^^^^^^^^^^^^^^^^^^ * To support Mac installations, use $(conda shell.bash activate geoips_conda) when activating conda vs "conda geoips_conda activate" * Allow use of GeoIPS-specific conda installation along-side user/system level installation where the user/system level installation may be initialized in `.bash_profile`. Uses GeoIPS-specific installation by default, if it is found. color_prompt ^^^^^^^^^^^^ * Add "$CONDA_PROMPT_MODIFIER" to $PS1 repo_clone_update_install.sh ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * If GEOIPS_TESTDATA_DIR, GEOIPS_PACKAGES_DIR, or GEOIPS_DEPENDENCIES_DIR are set, use those, otherwise default to placing under $GEOIPS BASEDIR * Update default branch from dev to main README.md ^^^^^^^^^ * Update github.com GEOIPS_ACTIVE_BRANCH from dev to main