geoips package#

Subpackages#

Submodules#

geoips.cli module#

GeoIPS

The Geolocated Information Processing System (GeoIPS) is a generalized processing system, providing a collection of algorithm and product implementations facilitating consistent and reliable application of specific products across a variety of sensors and data types.

GeoIPS acts as a toolbox for internal GeoIPS-based product development - all modules are expected to have simple inputs and outputs (Python numpy or dask arrays or xarrays, dictionaries, strings, lists), to enable portability and simplified interfacing between modules.

class geoips.cli.RawDescriptionArgumentDefaultsHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)[source]#

Bases: ArgumentDefaultsHelpFormatter, RawDescriptionHelpFormatter

Compound formatter class for user-readable help.

  • preserves the raw description formatting

  • adds defaults to helps.

geoips.cli.add_list_interface_parser(subparsers, name, aliases=None)[source]#

Add list interface parser.

geoips.cli.formclass#

alias of RawDescriptionArgumentDefaultsHelpFormatter

geoips.cli.get_interface(name)[source]#

Get interface.

geoips.cli.list_dev_interfaces()[source]#

Return a list of all developmental interfaces.

geoips.cli.list_interface_plugins(interface_name)[source]#

List interface plugins.

geoips.cli.list_interfaces(dev=False)[source]#

List interfaces.

geoips.cli.main()[source]#

Command line interface main function.

geoips.cli.print_table(title, headings, rows)[source]#

Print a column formatted table.

Parameters:

  • title (str) – A title for the table

  • headings (list of str) – A list of strings to use as column headings

  • rows (list of tupl of str) – A list of equal-length tuples

geoips.create_plugin_registries module#

Generates all available plugins from all installed GeoIPS packages.

After all plugins have been generated, they are written to a registered_plugins.json file which contains a dictionary of all the registered GeoIPS plugins across all plugin repositories.

Call ‘python create_plugin_registry.py’ to produce registered_plugins.json for EVERY currently installed plugin package. A separate registered_plugins.json is created at the top level package directory for each plugin package.

geoips.create_plugin_registries.add_module_plugin(package, relpath, plugins)[source]#

Add the module plugin associated with the filepaths and package to plugins.

Parameters:

  • package (str) – The current GeoIPS package being parsed

  • relpath (str) – The relpath path to the module plugin

  • plugins (dict) – A dictionary object of all installed GeoIPS package plugins

Returns:

error_message – String containing informative error messages from any plugins that were improperly formatted. An exception will be raised at the very end if error_message is not the empty string - this allows collecting ALL errors throughout the plugin registry process and reporting them all at once, to facilitate rapidly identifying and resolving errors.

Return type:

str

geoips.create_plugin_registries.add_text_plugin(package, relpath, plugins)[source]#

Add all txt plugins into plugin registries.

Parameters:

  • package (str) – The current GeoIPS package being parsed

  • relpath (str) – The relpath path to the module plugin

  • plugins (dict) – A dictionary object of all installed GeoIPS package plugins

Returns:

error_message – String containing informative error messages from any plugins that were improperly formatted. An exception will be raised at the very end if error_message is not the empty string - this allows collecting ALL errors throughout the plugin registry process and reporting them all at once, to facilitate rapidly identifying and resolving errors.

Return type:

str

geoips.create_plugin_registries.add_yaml_plugin(filepath, relpath, package, plugins)[source]#

Add the yaml plugin associated with the filepaths and package to plugins.

Parameters:

  • filepath (str) – The path of the plugin derived from resouces.files(package) / plugin

  • relpath (str) – The relative path to the filepath provided

  • package (str) – The current GeoIPS package being parsed

  • plugins (dict) – A dictionary object of all installed GeoIPS package plugins

Returns:

error_message – String containing informative error messages from any plugins that were improperly formatted. An exception will be raised at the very end if error_message is not the empty string - this allows collecting ALL errors throughout the plugin registry process and reporting them all at once, to facilitate rapidly identifying and resolving errors.

Return type:

str

geoips.create_plugin_registries.check_plugin_exists(package, plugins, interface_name, plugin_name, relpath)[source]#

Check if plugin already exists. If it does raise a PluginRegistryError.

Note this only checks for duplicate plugins within a single plugin package. The registry_sanity_check function is used after all plugins have been loaded to identify duplicate plugins across different plugin packages.

Parameters:

  • package (str) – The GeoIPS package being tested against

  • plugins (dict) – A dictionary object of all installed GeoIPS plugins in the current plugin package.

  • interface_name (str) – A string representing the GeoIPS interface being checked against

  • plugin_name (str) – A string representing the name of the plugin within the GeoIPS interface

Returns:

error_message – Empty string if no error, appropriate informative error message if duplicate plugin found in current plugin package.

Return type:

str

geoips.create_plugin_registries.create_plugin_registries(plugin_packages, save_type)[source]#

Generate all plugin paths associated with every installed GeoIPS packages.

These paths include schema plugins, module_based plugins and normal YAML plugins. After these paths are generated, they are sent to parse_plugin_paths, which generates and adds the actual plugins to the plugins dictionary.

Parameters:

  • plugin_packages (list EntryPoints) – A list of EntryPoints pointing to each installed GeoIPS package –> ie. [EntryPoint(name=’geoips’, value=’geoips’, group=’geoips.plugin_packages’), …]

  • save_type (str) – The file format to save to [json, yaml]

geoips.create_plugin_registries.format_docstring(docstring, use_regex=True)[source]#

Format the provided docstring placement in the plugin registry.

Found when using the CLI and inspecting the registry, some docstrings are formatted in a hard to read manner and look pretty bad. This function will format these docstrings to be easily readable, whether obtained via the CLI or manually inspected in the plugin registry.

Parameters:

  • docstring (str) –

    • The docstring which we are going to format.

  • use_regex (bool, optional (default=False)) –

    • Whether or not we want to apply regex formatting to the docstring. Usually recommended as it will replace ‘newline’ chars but not purposeful ‘.newline’ strings.

geoips.create_plugin_registries.get_parser()[source]#

Create the ArgumentParser for main.

geoips.create_plugin_registries.main()[source]#

Generate all available plugins from all installed GeoIPS packages.

After all plugins have been generated, they are written to registered_plugins.yaml containing a dictionary of all the registered GeoIPS plugins. Keys in this dictionary are the interface names, following by each plugin name.

Parameters:

args (list) – List of strings representing the arguments provided via command line.

geoips.create_plugin_registries.parse_plugin_paths(plugin_paths, package, package_dir, plugins)[source]#

Parse the plugin_paths provided from the current installed GeoIPS package.

Then, add them to the plugins dictionary based on the path of the plugin. The path contains information as to whether the plugin is a schema, module_based, or a normal yaml plugin.

Parameters:

  • plugin_paths (dict) – A dictionary of filepaths, with keys referring to the type of plugin

  • package (str) – The current GeoIPS package being parsed

  • package_dir (str) – The path to the current GeoIPS package (for determining relative paths)

  • plugins (dict) – A dictionary object of all installed GeoIPS package plugins

Returns:

error_message – String containing informative error messages from any plugins that were improperly formatted. An exception will be raised at the very end if error_message is not the empty string - this allows collecting ALL errors throughout the plugin registry process and reporting them all at once, to facilitate rapidly identifying and resolving errors.

Return type:

str

geoips.create_plugin_registries.registry_sanity_check(plugin_packages, save_type)[source]#

Check that each plugin package registry has no duplicate lowest depth entries.

If it does, raise a PluginRegistryError for that specific package, then remove all plugin registries from each package so the user must fix the error before continuing. While this doesn’t cause a normal error, duplicate plugins will be overwritten by same-named plugin found in the last package-entrypoint.

Parameters:

  • plugin_packages (list EntryPoints) – A list of EntryPoints pointing to each installed GeoIPS package –> ie. [EntryPoint(name=’geoips’, value=’geoips’, group=’geoips.plugin_packages’)]

  • save_type (str) – The file format to save to [json, yaml]

Returns:

  • No returns

  • Exceptions

  • ———-

  • PluginRegistryError – If error_message has contents, then raise PluginRegistryError(error_message). The error_message string will collect and report on all errors within this function prior to raising the PluginRegistryError to facilitate rapidly identifying and resolving errors throughout all plugin packages.

geoips.create_plugin_registries.remove_registries(plugin_packages)[source]#

Remove all plugin registries if a PluginRegistryError is raised.

Parameters:

plugin_packages (list EntryPoints) – A list of EntryPoints pointing to each installed GeoIPS package –> ie. [EntryPoint(name=’geoips’, value=’geoips’, group=’geoips.plugin_packages’), …]

Return type:

None

geoips.create_plugin_registries.write_registered_plugins(pkg_dir, plugins, save_type)[source]#

Write dictionary of all plugins available from installed GeoIPS packages.

Parameters:

  • pkg_dir (str) – Path in which to write registered_plugins

  • plugins (dict) – A dictionary object of all installed GeoIPS package plugins

  • save_type (str) – The file format to save to [json, yaml]

Return type:

No returns, file written to pkg_dir

geoips.errors module#

GeoIPS error module.

exception geoips.errors.CliError[source]#

Bases: Exception

Raise exception on command line interface error.

exception geoips.errors.CoverageError[source]#

Bases: Exception

Raise exception on data coverage error.

exception geoips.errors.MissingRequiredXarrayMetadata[source]#

Bases: Exception

Exception to be raised when required metadata missing from a geoips xarray.

Can be used from reader plugins if required metadata is missing, or from the procflows to validate the xarray output of plugins. At every stage, all GeoIPS xarrays should contain the required metadata.

exception geoips.errors.NoValidFilesError[source]#

Bases: Exception

Exception to be raised when there are no valid files found.

Can be used from reader plugins if no valid files were passed to the reader.

exception geoips.errors.OutputFormatterDatelineError[source]#

Bases: Exception

Exception to be raised when there is a dateline error in an output formatter.

exception geoips.errors.OutputFormatterInvalidProjectionError[source]#

Bases: Exception

Exception to be raised when there is a projection error in an OutputFormatter.

exception geoips.errors.PluginError[source]#

Bases: Exception

Exception to be raised when there is an error in a GeoIPS plugin.

exception geoips.errors.PluginPackageNotFoundError[source]#

Bases: Exception

Raise exception stating a certain plugin package was not found.

exception geoips.errors.PluginRegistryError[source]#

Bases: Exception

Exception to be raised when there is an error in a plugin registry.

geoips.geoips_utils module#

General high level utilities for geoips processing.

geoips.geoips_utils.copy_standard_metadata(orig_xarray, dest_xarray, extra_attrs=None, force=True)[source]#

Copy standard metadata from orig_xarray to dest_xarray.

Parameters:

  • orig_xarray (xarray.Dataset) – Original xarray to copy attributes from

  • dest_xarray (xarray.Dataset) – Destination xarray to copy attributes to

  • extra_attrs (list of str, optional) – Additional attributes to copy, beyond the standard metadata, by default None

  • force (bool, optional) – If force is True, overwrite existing attributes, by default True

Returns:

dest_xarray with standard metadata copied in place from orig_xarray.

Return type:

xarray.Dataset

geoips.geoips_utils.deprecation(message)[source]#

Print a deprecation warning during runtime.

geoips.geoips_utils.expose_geoips_commands(pkg_name=None, _test_log=None)[source]#

Expose a list of commands that operate in the GeoIPS environment.

Where, these commands are defined under ‘pyproject.toml:[tool.poetry.scripts]’, or ‘pyproject.toml:[project.entry-points.console_scripts]’

Parameters:

  • pkg_name (str (default = None)) –

    • The name of the GeoIPS Plugin Package whose command’s will be exposed.

    • If None, assume this was called via the commandline and retrieve package_name via that manner. Otherwise use the supplied package_name.

  • _test_log (logging.Logger (default = None)) –

    • If provided, use this logger instead. This is added as an optional argument so we can check the output of this command for our Unit Tests.

geoips.geoips_utils.find_all_txt_plugins(subdir='')[source]#

Find all txt plugins in registered plugin packages.

Search the plugins directory of each registered plugin package for files ending in .txt. Return list of files

geoips.geoips_utils.find_ascii_palette(name)[source]#

Find ASCII palette named “name”.

Search the plugins/txt/ascii_palettes directory for ASCII palettes to use as colormaps.

geoips.geoips_utils.get_numpy_seeded_random_generator()[source]#

Get a NumPy random generator seeded with a fixed value.

Returns:

A NumPy random generator initialized with a fixed seed value of 42.

Return type:

numpy.random.Generator

Notes

This function returns a seeded random generator using NumPy’s default_rng function with a fixed seed of 42. Using a fixed seed ensures that the random numbers generated by this generator will be reproducible across different runs. 42 was chosen because it is the answer to the universe and all things; that is to say: it’s selection is inconsequential and any other number could have been chosen.

Examples

>>> predictable_random = get_numpy_seeded_random_generator()
>>> predictable_random.integers(0, 10, size=5)
array([6, 3, 7, 4, 6])  # Example output, will be the same every time
geoips.geoips_utils.get_required_geoips_xarray_attrs()[source]#

Interface deprecated v2.0.

geoips.geoips_utils.is_editable(package_name)[source]#

Return whether or not ‘package_name’ has been installed in editable mode.

Where editable mode is a local package installed via ‘pip install -e <path_to_pkg> and non-editable mode is a local package installed via ‘pip install <pact_to_pkg>.

If the package under package_name doesn’t exist, raise a ValueError reporting that.

Parameters:

package_name (str) –

  • The name of the pip installed local package. (ie. “geoips”, “recenter_tc”, ..)

Returns:

editable –

  • The truth value as to whether or not the package was installed in editable mode.

Return type:

bool

geoips.geoips_utils.load_all_yaml_plugins()[source]#

Find all YAML plugins in registered plugin packages.

Search the plugins directory of each registered plugin package for files ending in .yaml. Read each plugin file

geoips.geoips_utils.merge_nested_dicts(dest, src, in_place=True, replace=False)[source]#

Perform an in-place merge of src into dest.

Performs an in-place merge of src into dest while preserving any values that already exist in dest.

geoips.geoips_utils.order_paths_from_least_to_most_specific(paths)[source]#

Orders a list of filesystem paths from least to most specific.

This function takes a list of filesystem paths and returns a new list of paths ordered from the least specific (higher-level directories) to the most specific (subdirectories and files). It expands environmental variables in paths.

Parameters:

paths (list of str or pathlib.Path) – A list of filesystem paths to be ordered.

Returns:

A list of filesystem paths ordered from least to most specific.

Return type:

list of pathlib.Path

Examples

>>> paths = [
...     '/home/user/docs/',
...     '/home/user/images/',
...     '/home/user/',
...     '/home/user/images/photo.jpg'
...     '/home/user/docs/report.txt',
... ]
>>> order_paths_from_least_to_most_specific(paths)
[PosixPath('/home/user/'),
 PosixPath('/home/user/docs/'),
 PosixPath('/home/user/images/'),
 PosixPath('/home/user/docs/report.txt'),
 PosixPath('/home/user/images/photo.jpg')]
geoips.geoips_utils.output_process_times(process_datetimes, num_jobs=None, job_str='GeoIPS 2')[source]#

Calculate and print the process times from the process_datetimes dictionary.

Parameters:

process_datetimes (dict) –

dictionary formatted as follows:

  • process_datetimes['overall_start'] - overall start datetime of the entire script

  • process_datetimes['overall_end'] - overall end datetime of the entire script

  • process_datetimes[process_name]['start'] - start time of an individual process

  • process_datetimes[process_name]['end'] - end time of an individual process

geoips.geoips_utils.remove_unsupported_kwargs(module, requested_kwargs)[source]#

Remove unsupported keyword arguments.

geoips.geoips_utils.replace_geoips_paths(path, replace_paths=None, base_paths=None, curly_braces=False)[source]#

Replace specified sub-paths in path with related environment variable names.

This function replaces paths in the provided path with their corresponding environment variable names. This is useful for generating output paths or metadata that are independent of specific installation directories.

For example, it can replace:

  • '/home/user/geoproc/geoips_packages' with '$GEOIPS_PACKAGES_DIR'

  • '/home/user/geoproc/geoips_outdirs' with '$GEOIPS_OUTDIRS'

  • '/home/user/geoproc' with '$GEOIPS_BASEDIR'

Parameters:

  • path (str or pathlib.Path) – The path in which to replace base paths.

  • replace_paths (list of str, optional) –

    A list of environment variable names whose corresponding paths should be replaced in path. If None, defaults to:

    ['$GEOIPS_OUTDIRS', '$GEOIPS_PACKAGES_DIR', '$GEOIPS_TESTDATA_DIR', '$GEOIPS_DEPENDENCIES_DIR', '$GEOIPS_BASEDIR']

  • base_paths (dict, optional) – A dictionary mapping environment variable names to their corresponding base paths. If None, defaults to geoips.filenames.base_paths.PATH.

  • curly_braces (bool, default=False) – If True, includes curly braces in the environment variables (e.g., '${GEOIPS_BASEDIR}'), otherwise excludes them (e.g., '$GEOIPS_BASEDIR').

Returns:

The path with specified base paths replaced with environment variable names.

Return type:

str

Notes

The function iterates over the provided replace_paths in reverse order (from most specific to least specific) and replaces the first matching base path in the given path with the corresponding environment variable name.

Examples

>>> path = '/home/user/geoproc/geoips_packages/module/file.py'
>>> base_paths = {
...     'GEOIPS_PACKAGES_DIR': '/home/user/geoproc/geoips_packages',
...     'GEOIPS_BASEDIR': '/home/user/geoproc'
... }
>>> replace_geoips_paths(path, base_paths=base_paths)
'$GEOIPS_PACKAGES_DIR/module/file.py'
geoips.geoips_utils.replace_geoips_paths_in_dict(replace_dict, replace_paths=None, base_paths=None, curly_braces=False)[source]#

Replace geoips paths in every path-based element within a dictionary.

geoips.geoips_utils.replace_geoips_paths_in_list(replace_list, replace_paths=None, base_paths=None, curly_braces=False)[source]#

Replace GeoIPS paths with geoips settings in elements of a list.

This function iterates over each element in the provided replace_list, attempting to replace GeoIPS paths within each element using the replace_geoips_paths function. If an element raises a TypeError when cast to a pathlib path, it is skipped.

Parameters:

  • replace_list (list) – A list of elements to process. Elements can be of any type, but only those that are Path-like will be processed.

  • replace_paths (dict, optional) – Passed to replace_geoips_paths

  • base_paths (dict, optional) – Passed to replace_geoips_paths

  • curly_braces (bool, optional) – Passed to replace_geoips_paths

Returns:

A new list containing the elements with GeoIPS paths replaced where possible. Elements that could not be processed are included unchanged.

Return type:

list

Examples

>>> replace_geoips_paths_in_list(['/home/geoips/data/project',
... 'no_replacement_here'])
['$GEOIPS_DATA_DIR/project', 'no_replacement_here']

See also

replace_geoips_paths

Function used to replace GeoIPS paths in individual elements.

geoips.plugin_registry module#

PluginRegistry class to interface with the JSON plugin registries.

The “create_plugin_registries” utility generates a JSON file at the top level of every geoips plugin package with a complete list of all plugins with the associated metadata (everything except the actual contents of the plugin itself).

Once all of the registered_plugins.json files have been generated via create_plugin_registries, this class uses those registries to quickly identify and open plugins as required. Previously the individual interface classes would open all plugins every time one was required, so moving this process into a single PluginRegistry object allows us to more effectively cache plugins across all interfaces, and avoid reading in all plugins multiple times.

class geoips.plugin_registry.PluginRegistry(_test_registry_files=[])[source]#

Bases: object

Plugin Registry class definition.

Represents all of the plugins found in all of the available GeoIPS packages. This class will load a plugin when requested, rather than loading all plugins when GeoIPS is instantiated.

get_interface_plugin_info(interface)[source]#

Find an interface in the registry and return its corresponding info.

This should remove all plugin loading from the base interfaces and allow us to only load one plugin at a time

get_interface_plugin_names(interface)[source]#

List all available plugins in the an interface of the registry.

This should remove all plugin loading from the base interfaces and allow us to only load one plugin at a time

get_plugin_info(interface, plugin_name)[source]#

Find a plugin in the registry and return its info.

This should remove all plugin loading from the base interfaces and allow us to only load one plugin at a time

identify_plugin_type(interface)[source]#

Identify the Plugin Type based on the provided interface.

property interface_mapping#

Plugin Registry interface_mapping attribute.

list_plugins(interface)[source]#

List the plugins available for an interface ONLY based on the registries.

This should not load any plugins, just return info from the registries.

property registered_plugins#

Plugin Registry registered_plugins attribute.

Module contents#

The Geolocated Information Processing System (GeoIPS).

GeoIPS ® Base Package#

The GeoIPS Base Package provides a Python 3 based architecture supporting a wide variety of satellite and weather data processing. The modular nature of the GeoIPS base infrastructure also allows plug-and-play capability for user-specified custom functionality.

Homepage: NRLMMD-GEOIPS/geoips

On this page

This Page