geoips package

Subpackages

• geoips.commandline package
  • Submodules
  • geoips.commandline.args module
    • add_args()
    • check_command_line_args()
    • get_command_line_args()
  • geoips.commandline.list_available_modules module
    • main()
  • geoips.commandline.log_setup module
    • setup_logging()
  • geoips.commandline.run_procflow module
    • main()
  • geoips.commandline.test_interfaces module
    • main()
  • geoips.commandline.update_tc_tracks_database module
    • main()
  • Module contents
• geoips.data_manipulations package
  • Submodules
  • geoips.data_manipulations.conversions module
    • unit_conversion()
  • geoips.data_manipulations.corrections module
    • apply_data_range()
    • apply_gamma()
    • apply_maximum_value()
    • apply_minimum_value()
    • apply_offset()
    • apply_scale_factor()
    • apply_solar_zenith_correction()
    • invert_data_range()
    • mask_day()
    • mask_night()
    • normalize()
  • geoips.data_manipulations.info module
    • percent_not_nan()
    • percent_unmasked()
  • geoips.data_manipulations.merge module
    • daterange()
    • find_datafiles_in_range()
    • get_matching_files()
    • hourrange()
    • minrange()
  • Module contents
• geoips.dev package
  • Submodules
  • geoips.dev.output_config module
    • get_filename_formatter_kwargs()
    • get_filename_formatters()
    • get_metadata_filename_formatter()
    • get_metadata_filename_formatter_kwargs()
    • get_metadata_output_formatter()
    • get_metadata_output_formatter_kwargs()
    • get_minimum_coverage()
    • get_output_config_type()
    • get_output_formatter()
    • get_output_formatter_kwargs()
    • is_valid_output_config()
    • produce_current_time()
    • set_lonlat_spacing()
    • test_output_config_interface()
  • geoips.dev.product module
    • get_cmap_from_product()
    • get_covg_args_from_product()
    • get_covg_from_product()
    • get_data_range()
    • get_product_display_name()
    • get_requested_datasets_for_variables()
    • get_required_variables()
  • Module contents
• geoips.filenames package
  • Submodules
  • geoips.filenames.base_paths module
    • make_dirs()
  • geoips.filenames.duplicate_files module
    • remove_duplicates()
  • Module contents
• geoips.image_utils package
  • Submodules
  • geoips.image_utils.colormap_utils module
    • create_linear_segmented_colormap()
    • from_ascii()
    • set_matplotlib_colors_rgb()
    • set_matplotlib_colors_standard()
    • set_mpl_colors_info_dict()
  • geoips.image_utils.maps module
    • check_feature_annotator()
    • check_gridline_annotator()
    • compute_lat_auto_spacing()
    • compute_lon_auto_spacing()
    • draw_features()
    • draw_gridlines()
    • ellps2axis()
    • is_crs()
    • meridians()
    • parallels()
    • set_gridlines_info_dict()
  • geoips.image_utils.mpl_utils module
    • alpha_from_masked_arrays()
    • create_colorbar()
    • create_figure_and_main_ax_and_mapobj()
    • get_title_string_from_objects()
    • percent_unmasked_rgba()
    • plot_image()
    • plot_overlays()
    • remove_duplicates()
    • rgba_from_arrays()
    • save_image()
    • set_fonts()
    • set_title()
  • Module contents
• geoips.interfaces package
  • Subpackages
    • geoips.interfaces.module_based package
      • Submodules
      • geoips.interfaces.module_based.algorithms module
      • geoips.interfaces.module_based.colormappers module
      • geoips.interfaces.module_based.coverage_checkers module
      • geoips.interfaces.module_based.filename_formatters module
      • geoips.interfaces.module_based.interpolators module
      • geoips.interfaces.module_based.output_formatters module
      • geoips.interfaces.module_based.procflows module
      • geoips.interfaces.module_based.readers module
      • geoips.interfaces.module_based.sector_adjusters module
      • geoips.interfaces.module_based.sector_metadata_generators module
      • geoips.interfaces.module_based.sector_spec_generators module
      • geoips.interfaces.module_based.title_formatters module
      • Module contents
    • geoips.interfaces.yaml_based package
      • Submodules
      • geoips.interfaces.yaml_based.feature_annotators module
      • geoips.interfaces.yaml_based.gridline_annotators module
      • geoips.interfaces.yaml_based.product_defaults module
      • geoips.interfaces.yaml_based.products module
      • geoips.interfaces.yaml_based.sectors module
      • Module contents
  • Submodules
  • geoips.interfaces.base module
    • BaseInterface
    • BaseModuleInterface
      • BaseModuleInterface.get_plugin()
      • BaseModuleInterface.get_plugins()
      • BaseModuleInterface.plugin_is_valid()
      • BaseModuleInterface.plugins_all_valid()
      • BaseModuleInterface.test_interface()
    • BaseModulePlugin
    • BaseYamlInterface
      • BaseYamlInterface.get_plugin()
      • BaseYamlInterface.get_plugins()
      • BaseYamlInterface.plugin_is_valid()
      • BaseYamlInterface.plugins_all_valid()
      • BaseYamlInterface.test_interface()
      • BaseYamlInterface.validator
    • BaseYamlPlugin
    • YamlPluginValidator
      • YamlPluginValidator.schemas
      • YamlPluginValidator.validate()
      • YamlPluginValidator.validate_list()
      • YamlPluginValidator.validators
    • extend_with_default()
    • get_schemas()
    • get_validators()
    • plugin_module_to_obj()
    • plugin_repr()
    • plugin_yaml_to_obj()
  • Module contents
• geoips.plugins package
  • Subpackages
    • geoips.plugins.modules package
      • Subpackages
      • Module contents
  • Module contents
• geoips.sector_utils package
  • Submodules
  • geoips.sector_utils.estimate_area_extent module
    • center_longitude()
    • convert_west2east()
    • esitmate_area_from_center()
    • estimate_area_extent()
    • generateMinMaxLatLong()
    • haversine_distance()
  • geoips.sector_utils.overpass_predictor module
    • calculate_overpass()
    • check_tle_name_to_passed_names()
    • floor_minute()
    • predict_overpass_area_def()
    • predict_overpass_yaml()
    • predict_satellite_overpass()
    • read_satellite_tle()
  • geoips.sector_utils.projections module
    • get_projection()
  • geoips.sector_utils.tc_tracks module
    • create_tc_sector_info_dict()
    • get_tc_area_id()
    • get_tc_long_description()
    • interpolate_storm_location()
    • set_tc_area_def()
    • trackfile_to_area_defs()
  • geoips.sector_utils.tc_tracks_database module
    • check_db()
    • get_all_storms_from_db()
    • open_tc_db()
    • reprocess_storm()
    • update_fields()
  • geoips.sector_utils.utils module
    • check_center_coverage()
    • copy_sector_info()
    • create_areadefinition_from_yaml()
    • filter_area_defs_actual_time()
    • get_lat_center()
    • get_lon_center()
    • get_max_lat()
    • get_max_lon()
    • get_min_lat()
    • get_min_lon()
    • get_sectors_from_yamls()
    • get_static_area_defs_for_xarray()
    • get_tc_area_defs_for_xarray()
    • get_trackfile_area_defs()
    • is_dynamic_sector()
    • is_requested_aid_type()
    • is_sector_type()
    • remove_duplicate_storm_positions()
    • set_tc_coverage_check_area_def()
    • set_text_area_def()
    • storm_locations_match()
  • geoips.sector_utils.yaml_utils module
    • add_description_to_yamldict()
    • add_dynamic_datetime_to_yamldict()
    • add_projection_to_yamldict()
    • add_sectorinfo_to_yamldict()
    • area_def_to_yamldict()
    • area_def_to_yamlfile()
    • write_yamldict()
  • Module contents
• geoips.utils package
  • Submodules
  • geoips.utils.decorators module
    • deprecated
    • developmental()
  • geoips.utils.memusg module
    • print_mem_usage()
    • print_resource_usage()
  • Module contents
• geoips.xarray_utils package
  • Submodules
  • geoips.xarray_utils.data module
    • get_lat_lon_points()
    • get_lat_lon_points_numpy()
    • get_sectored_xarrays()
    • get_vis_ir_bg()
    • sector_xarray_dataset()
    • sector_xarray_spatial()
    • sector_xarray_temporal()
    • sector_xarrays()
  • geoips.xarray_utils.time module
    • get_datetime_from_datetime64()
    • get_max_from_xarray_time()
    • get_min_from_xarray_time()
    • get_posix_from_datetime()
  • Module contents

Submodules

geoips.cli module

GeoIPS

The Geolocated Information Processing System (GeoIPS) is a generalized processing system, providing a collection of algorithm and product implementations facilitating consistent and reliable application of specific products across a variety of sensors and data types.

GeoIPS acts as a toolbox for internal GeoIPS-based product development - all modules are expected to have simple inputs and outputs (Python numpy or dask arrays or xarrays, dictionaries, strings, lists), to enable portability and simplified interfacing between modules.

class geoips.cli.RawDescriptionArgumentDefaultsHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)

Bases: ArgumentDefaultsHelpFormatter, RawDescriptionHelpFormatter

Compound formatter class for user-readable help.

• preserves the raw description formatting

• adds defaults to helps.

geoips.cli.add_list_interface_parser(subparsers, name, aliases=None)

Add list interface parser.

geoips.cli.formclass

alias of RawDescriptionArgumentDefaultsHelpFormatter

geoips.cli.get_interface(name)

Get interface.

geoips.cli.list_dev_interfaces()

Return a list of all developmental interfaces.

geoips.cli.list_interface_plugins(interface_name)

List interface plugins.

geoips.cli.list_interfaces(dev=False)

List interfaces.

geoips.cli.main()

Command line interface main function.

geoips.cli.print_table(title, headings, rows)

Print a column formatted table.

Parameters:

• title (str) – A title for the table

• headings (list of str) – A list of strings to use as column headings

• rows (list of tupl of str) – A list of equal-length tuples

geoips.compare_outputs module

Test script for representative product comparisons.

geoips.compare_outputs.compare_outputs(compare_path, output_products, test_product_func=None)

Compare the “correct” imagery found the list of current output_products.

Compares files produced in the current processing run with the list of “correct” files contained in “compare_path”.

Parameters:

• compare_path (str) – Path to directory of “correct” products - filenames must match output_products

• output_products (list of str) – List of strings of current output products, to compare with products in compare_path

• test_product_func (function, default=None) –

  Alternative function to be used for testing output product

  • Call signature must be:

    • output_product, compare_product, goodcomps, badcomps, compare_strings

  • Return must be:

    • goodcomps, badcomps, compare_strings

  • If None, use geoips.compare_outputs.test_product)

Returns:

Binary code: 0 if all comparisons were completed successfully.

Return type:

int

geoips.compare_outputs.geoips_netcdf_match(output_product, compare_product)

Check if two geoips formatted netcdf files match.

Parameters:

• output_product (str) – Full path to current output product

• compare_product (str) – Full path to comparison product

Returns:

Return True if products match, False if they differ

Return type:

bool

geoips.compare_outputs.geotiffs_match(output_product, compare_product)

Use diff system command to compare currently produced image to correct image.

Parameters:

• output_product (str) – Full path to current output product

• compare_product (str) – Full path to comparison product

Returns:

Return True if images match, False if they differ

Return type:

bool

geoips.compare_outputs.get_out_diff_fname(compare_product, output_product, ext=None, flag=None)

Obtain the filename for output and comparison product diff.

Parameters:

• compare_product (str) – Full path to product filename in the comparison directory

• output_product (str) – Full path to product filename in the current output directory

• ext (str, default=None) – Extension to use as an alternative to the original file extension

• flag (str, default=None) – Additional identifying string to include in output diff filename

Returns:

out_diff_fname – Full path to output diff file.

Return type:

str

geoips.compare_outputs.gunzip_product(fname)

Gunzip file fname.

Parameters:

fname (str) – File to gunzip.

Returns:

Filename after gunzipping

Return type:

str

geoips.compare_outputs.gzip_product(fname)

Gzip file fname.

Parameters:

fname (str) – File to gzip.

Returns:

Filename after gzipping

Return type:

str

geoips.compare_outputs.images_match(output_product, compare_product, fuzz='5%')

Use imagemagick compare system command to compare two images.

Parameters:

• output_product (str) – Current output product

• compare_product (str) – Path to comparison product

• fuzz (str, optional) – “fuzz” argument to pass to compare - larger “fuzz” factor to make comparison less strict, by default 5%.

Returns:

Return True if images match, False if they differ

Return type:

bool

geoips.compare_outputs.is_geoips_netcdf(fname)

Check if fname is a geoips formatted netcdf file.

Parameters:

fname (str) – Name of file to check.

Returns:

True if it is a geoips netcdf file, False otherwise.

Return type:

bool

geoips.compare_outputs.is_geotiff(fname)

Determine if fname is a geotiff file.

Parameters:

fname (str) – Name of file to check.

Returns:

True if it is a geotiff file, False otherwise.

Return type:

bool

geoips.compare_outputs.is_gz(fname)

Check if fname is a gzip file.

Parameters:

fname (str) – Name of file to check.

Returns:

True if it is a gz file, False otherwise.

Return type:

bool

geoips.compare_outputs.is_image(fname)

Determine if fname is an image file.

Parameters:

fname (str) – Name of file to check.

Returns:

True if it is an image file, False otherwise.

Return type:

bool

geoips.compare_outputs.is_text(fname)

Check if fname is a text file.

Parameters:

fname (str) – Name of file to check.

Returns:

True if it is a text file, False otherwise.

Return type:

bool

geoips.compare_outputs.print_gunzip_to_file(fobj, gunzip_fname)

Write the command to gunzip the passed “gunzip_fname” to file.

Writes to the currently open file object, if required.

geoips.compare_outputs.print_gzip_to_file(fobj, gzip_fname)

Write the command to gzip the passed “gzip_fname” to file.

Writes to the currently open file object, if required.

geoips.compare_outputs.test_product(output_product, compare_product, goodcomps, badcomps, compare_strings)

Test output_product against “good” product stored in “compare_path”.

Parameters:

• output_product (str) –

  • Full path to current output product

• compare_product (str) –

  • Full path to “good” comparison product

• goodcomps (list of str) –

  • List of full paths to all “good” successful comparisons (output and compare images match)

  • Each str is prepended with a “compare_string” tag to identify which comparison type was performed.

• badcomps (list of str) –

  • List of full paths to all “bad” unsuccessful comparisons (output and compare images differ)

  • Each str is prepended with a “compare_string” tag to identify which comparison type was performed.

• compare_strings (list of str) –

  • List of all comparison “tags” included in goodcomps and badcomps lists.

  • This list is used to remove the comparison tags from goodcomps and badcomps to retrieve only the file path.

Returns:

• goodcomps (list of str) – All current good comparisons appended to the list passed in.

• badcomps (list of str) – All current bad comparisons appended to the list passed in.

• compare_strings (list of str) – All current comparison “tags” added to the list passed in.

Raises:

TypeError – Raised when current output product does not have an associated comparison test defined.

geoips.compare_outputs.text_match(output_product, compare_product)

Check if two text files match.

Parameters:

• output_product (str) – Full path to current output product

• compare_product (str) – Full path to “good” comparison product

Returns:

Return True if products match, False if they differ

Return type:

bool

geoips.errors module

GeoIPS error module.

exception geoips.errors.CoverageError

Bases: Exception

Raise exception on data coverage error.

exception geoips.errors.EntryPointError

Bases: Exception

Exception to be raised when an entry-point cannot be found.

exception geoips.errors.PluginError

Bases: Exception

Exception to be raised when there is an error in a plugin module.

geoips.geoips_utils module

General high level utilities for geoips processing.

geoips.geoips_utils.copy_standard_metadata(orig_xarray, dest_xarray, extra_attrs=None, force=True)

Copy standard metadata from orig_xarray to dest_xarray.

Parameters:

• orig_xarray (xarray.Dataset) – Original xarray to copy attributes from

• dest_xarray (xarray.Dataset) – Destination xarray to copy attributes to

• extra_attrs (list of str, optional) – Additional attributes to copy, beyond the standard metadata, by default None

• force (bool, optional) – If force is True, overwrite existing attributes, by default True

Returns:

dest_xarray with standard metadata copied in place from orig_xarray.

Return type:

xarray.Dataset

geoips.geoips_utils.deprecation(message)

Print a deprecation warning during runtime.

geoips.geoips_utils.find_all_txt_plugins(subdir='')

Find all txt plugins in registered plugin packages.

Search the plugins directory of each registered plugin package for files ending in .txt. Return list of files

geoips.geoips_utils.find_ascii_palette(name)

Find ASCII palette named “name”.

Search the plugins/txt/ascii_palettes directory for ASCII palettes to use as colormaps.

geoips.geoips_utils.find_config(subpackage_name, config_basename, txt_suffix='.yaml')

Find matching config file within GEOIPS packages.

Given ‘subpackage_name’, ‘config_basename’, and txt_suffix, find matching text file within GEOIPS packages.

Parameters:

• subpackage_name (str) – subdirectory under GEOIPS package to look for text file ie text_fname = geoips/<subpackage_name>/<config_basename><txt_suffix>

• config_basename (str) – text basename to look for, ie text_fname = geoips/<subpackage_name>/<config_basename><txt_suffix>

• txt_suffix (str) – suffix to look for on config file, defaults to “.yaml” ie text_fname = geoips/<subpackage_name>/<config_basename><txt_suffix>

Returns:

text_fname – Full path to text filename

Return type:

str

geoips.geoips_utils.find_entry_point(namespace, name, default=None)

Find object matching ‘name’ using GEOIPS entry point namespace ‘namespace’.

Automatically add ‘geoips’ prefix to namespace for disambiguation.

Parameters:

• namespace (str) – Entry point namespace (e.g. ‘readers’)

• name (str) – Entry point name (e.g. ‘amsr2_netcdf’)

• default (entry point, optional) – Default value if no match is found. If this is not set (i.e. None), then no match will result in an exception

geoips.geoips_utils.get_all_entry_points(namespace)

Return all entry points in GEOIPS entry point namespace ‘namespace’.

Automatically add ‘geoips’ prefix to namespace for disambiguation.

Parameters:

namespace (str) – Entry point namespace (e.g. ‘readers’)

geoips.geoips_utils.get_entry_point_group(group)

Get entry point group.

geoips.geoips_utils.get_required_geoips_xarray_attrs()

Interface deprecated v2.0.

geoips.geoips_utils.list_entry_points(namespace)

List names of objects in GEOIPS entry point namespace ‘namespace’.

Automatically add ‘geoips’ prefix to namespace for disambiguation.

Parameters:

namespace (str) – Entry point namespace (e.g. ‘readers’)

geoips.geoips_utils.list_product_source_dict_yamls()

List all YAML files containing product source specifications.

Search in all geoips packages.

Returns:

List of all product source dict YAMLs in all geoips packages

Return type:

list

geoips.geoips_utils.list_product_specs_dict_yamls()

List all YAML files containing product params in all geoips packages.

Returns:

List of all product params dict YAMLs in all geoips packages

Return type:

list

geoips.geoips_utils.load_all_yaml_plugins()

Find all YAML plugins in registered plugin packages.

Search the plugins directory of each registered plugin package for files ending in .yaml. Read each plugin file

geoips.geoips_utils.merge_nested_dicts(dest, src, in_place=True)

Perform an in-place merge of src into dest.

Performs an in-place merge of src into dest while preserving any values that already exist in dest.

geoips.geoips_utils.output_process_times(process_datetimes, num_jobs=None, job_str='GeoIPS 2')

Calculate and print the process times from the process_datetimes dictionary.

Parameters:

process_datetimes (dict) –

dictionary formatted as follows:

• process_datetimes['overall_start'] - overall start datetime of the entire script

• process_datetimes['overall_end'] - overall end datetime of the entire script

• process_datetimes[process_name]['start'] - start time of an individual process

• process_datetimes[process_name]['end'] - end time of an individual process

geoips.geoips_utils.replace_geoips_paths(fname, replace_paths=None, base_paths=None)

Replace standard environment variables with their non-expanded equivalents.

Ie, replace

• $HOME/geoproc/geoips_packages with $GEOIPS_PACKAGES_DIR

• $HOME/geoproc/geoips_outdirs with $GEOIPS_OUTDIRS

• $HOME/geoproc with $GEOIPS_BASEDIR

This allows generating output YAML fields / NetCDF attributes that can match between different instantiations.

Parameters:

• fname (str) – Full path to a filename on disk

• replace_paths (list, default=None) –

  • Explicit list of standard variable names you would like replaced.

  • If None, replace ['GEOIPS_OUTDIRS', 'GEOIPS_PACKAGES_DIR', 'GEOIPS_TESTDATA_DIR', 'GEOIPS_DEPENDENCIES_DIR', 'GEOIPS_BASEDIR']

• base_paths (list, default=None) –

  • List of PATHS dictionaries in which to find the “replace_paths” variables

  • If None, use geoips.filenames.base_paths

Returns:

fname – Path to file on disk, with explicit path replaced with environment variable name and/or full URL.

Return type:

str

Notes

Note it replaces ALL standard variables that have a corresponding <key>_URL variable.

Additionally, it replaces variables specified in “replace_paths” list with the unexpanded environment variable name.

Module contents

The Geolocated Information Processing System (GeoIPS).

GeoIPS ® Base Package

The GeoIPS Base Package provides a Python 3 based architecture supporting a wide variety of satellite and weather data processing. The modular nature of the GeoIPS base infrastructure also allows plug-and-play capability for user-specified custom functionality.

Homepage: NRLMMD-GEOIPS/geoips