geoips package#

Subpackages#

Submodules#

geoips.cli module#

GeoIPS

The Geolocated Information Processing System (GeoIPS) is a generalized processing system, providing a collection of algorithm and product implementations facilitating consistent and reliable application of specific products across a variety of sensors and data types.

GeoIPS acts as a toolbox for internal GeoIPS-based product development - all modules are expected to have simple inputs and outputs (Python numpy or dask arrays or xarrays, dictionaries, strings, lists), to enable portability and simplified interfacing between modules.

class geoips.cli.RawDescriptionArgumentDefaultsHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)[source]#

Bases: ArgumentDefaultsHelpFormatter, RawDescriptionHelpFormatter

Compound formatter class for user-readable help.

  • preserves the raw description formatting

  • adds defaults to helps.

geoips.cli.add_list_interface_parser(subparsers, name, aliases=None)[source]#

Add list interface parser.

geoips.cli.formclass#

alias of RawDescriptionArgumentDefaultsHelpFormatter

geoips.cli.get_interface(name)[source]#

Get interface.

geoips.cli.list_dev_interfaces()[source]#

Return a list of all developmental interfaces.

geoips.cli.list_interface_plugins(interface_name)[source]#

List interface plugins.

geoips.cli.list_interfaces(dev=False)[source]#

List interfaces.

geoips.cli.main()[source]#

Command line interface main function.

geoips.cli.print_table(title, headings, rows)[source]#

Print a column formatted table.

Parameters:

  • title (str) – A title for the table

  • headings (list of str) – A list of strings to use as column headings

  • rows (list of tupl of str) – A list of equal-length tuples

geoips.constants module#

Scientific and data-processing constants used across GeoIPS.

geoips.errors module#

GeoIPS error module.

exception geoips.errors.BoundaryIOError[source]#

Bases: GeoipsError

Reader or output_formatter I/O failure.

Raised when a boundary step (reader or output_formatter) fails to read from or write to disk.

exception geoips.errors.CliError[source]#

Bases: Exception

Raise exception on command line interface error.

exception geoips.errors.ConfigError[source]#

Bases: GeoipsError

Raise exception on GeoIPS configuration error.

Used for invalid configuration files, plugin config registration problems, and environment-variable collisions.

exception geoips.errors.CoverageError[source]#

Bases: GeoipsError

Raise exception on data coverage error.

exception geoips.errors.DataTreeSchemaError[source]#

Bases: GeoipsError

Required attributes or child nodes are missing from a DataTree.

Raised by the workflow runner when an output DataTree fails structural invariants (e.g., a step node missing required attrs).

exception geoips.errors.DependencyCycleError[source]#

Bases: GeoipsError

The depends_on graph contains a directed cycle.

Raised during workflow validation when steps form a circular dependency chain that cannot be topologically sorted.

exception geoips.errors.FileFormatError[source]#

Bases: ValueError, GeoipsError

Exception to be raised when a file is badly formatted and cannot be used.

exception geoips.errors.GeoipsError[source]#

Bases: Exception

Base class for all GeoIPS-specific exceptions.

Catch this to handle any GeoIPS-originated error in a single except clause.

exception geoips.errors.MissingRequiredXarrayMetadata[source]#

Bases: KeyError, GeoipsError

Exception to be raised when required metadata missing from a geoips xarray.

Can be used from reader plugins if required metadata is missing, or from the procflows to validate the xarray output of plugins. At every stage, all GeoIPS xarrays should contain the required metadata.

exception geoips.errors.NoValidFilesError[source]#

Bases: OSError, GeoipsError

Exception to be raised when there are no valid files found.

Can be used from reader plugins if no valid files were passed to the reader.

exception geoips.errors.OutputFormatterDatelineError[source]#

Bases: ValueError, GeoipsError

Exception to be raised when there is a dateline error in an output formatter.

exception geoips.errors.OutputFormatterInvalidProjectionError[source]#

Bases: ValueError, GeoipsError

Exception to be raised when there is a projection error in an OutputFormatter.

exception geoips.errors.PluginPackageNotFoundError[source]#

Bases: ModuleNotFoundError, GeoipsError

Raise exception stating a certain plugin package was not found.

exception geoips.errors.PluginResolutionError[source]#

Bases: GeoipsError

A plugin name: reference in a step cannot be resolved.

Raised when PluginRegistry.get_plugin(kind, name) returns no result for a given (kind, name) pair referenced in a workflow YAML.

exception geoips.errors.RetentionConfigError[source]#

Bases: GeoipsError

Conflicting or invalid retention settings.

Raised when a workflow’s retention policy combined with per-step keep flags creates an unresolvable configuration.

exception geoips.errors.WorkflowSpecError[source]#

Bases: GeoipsError

Workflow YAML fails pydantic validation.

Raised when the YAML structure, field types, or required values do not conform to the workflow specification.

geoips.geoips_utils module#

General high level utilities for geoips processing.

geoips.geoips_utils.call_cmd(cmd, output_log_fname=None, stdout_newline_replace_val=None, use_logging=True, use_print=False, pipe=False, cwd=None)[source]#

Call command using subprocess.run, and log to specified log file if requested.

geoips.geoips_utils.call_cmd_pipe(cmd, output_log_fname=None, stdout_newline_replace_val=None, use_logging=True, use_print=False, cwd=None)[source]#

Call command using Popen, and log to specified log file if requested.

geoips.geoips_utils.copy_standard_metadata(orig_xarray, dest_xarray, extra_attrs=None, force=True)[source]#

Copy standard metadata from orig_xarray to dest_xarray.

The values automatically copied within this method should match the required and optional metadata specified in the GeoIPS xarray_standards documentation.

Parameters:

  • orig_xarray (xarray.Dataset) – Original xarray to copy attributes from

  • dest_xarray (xarray.Dataset) – Destination xarray to copy attributes to

  • extra_attrs (list of str, optional) – Additional attributes to copy, beyond the standard metadata, by default None

  • force (bool, optional) – If force is True, overwrite existing attributes, by default True

Returns:

dest_xarray with standard metadata copied in place from orig_xarray.

Return type:

xarray.Dataset

geoips.geoips_utils.deprecation(message)[source]#

Print a deprecation warning during runtime.

geoips.geoips_utils.expose_geoips_commands(pkg_name=None, _test_log=None, column_width=None)[source]#

Expose a list of commands that operate in the GeoIPS environment.

Where, these commands are defined under ‘pyproject.toml:[tool.poetry.scripts]’, or ‘pyproject.toml:[project.entry-points.console_scripts]’

Parameters:

  • pkg_name (str (default = None)) –

    • The name of the GeoIPS Plugin Package whose command’s will be exposed.

    • If None, assume this was called via the commandline and retrieve package_name via that manner. Otherwise use the supplied package_name.

  • _test_log (logging.Logger (default = None)) –

    • If provided, use this logger instead. This is added as an optional argument so we can check the output of this command for our Unit Tests.

  • column_width (int (default = None)) –

    • If provided, use explicit column width for table. By default, use get_terminal_size().columns // 2

geoips.geoips_utils.find_all_txt_plugins(subdir='')[source]#

Find all txt plugins in registered plugin packages.

Search the plugins directory of each registered plugin package for files ending in .txt. Return list of files

geoips.geoips_utils.find_ascii_palette(name)[source]#

Find ASCII palette named “name”.

Search the plugins/txt/ascii_palettes directory for ASCII palettes to use as colormaps.

geoips.geoips_utils.get_interface_module(namespace)[source]#

Retrieve the interface module from a given namespace.

Since this function uses the first portion of a namespace, I.e. (geoips.plugin_packages –> geoips), only interfaces implemented in that exact package will be recognized in that namespace. This means that if developers or users implement new interfaces in the ‘geoips.plugin_packages’ namespace, they still won’t be recognized or available for use in GeoIPS.

To get new interfaces to work, you must implement them in a separate namespace that is named the same as the plugin package that implemented them. I.e. (splunk.plugin_packages –> splunk). Any other package that falls under that namespace can make use of these interfaces, as well as GeoIPS’ interfaces.

Parameters:

namespace (str) –

  • The namespace containg the requested interface module. I.e. ‘geoips.plugin_packages’.

geoips.geoips_utils.get_numpy_seeded_random_generator()[source]#

Get a NumPy random generator seeded with a fixed value.

Returns:

A NumPy random generator initialized with a fixed seed value of 42.

Return type:

numpy.random.Generator

Notes

This function returns a seeded random generator using NumPy’s default_rng function with a fixed seed of 42. Using a fixed seed ensures that the random numbers generated by this generator will be reproducible across different runs. 42 was chosen because it is the answer to the universe and all things; that is to say: it’s selection is inconsequential and any other number could have been chosen.

Examples

>>> predictable_random = get_numpy_seeded_random_generator()
>>> predictable_random.integers(0, 10, size=5)
array([6, 3, 7, 4, 6])  # Example output, will be the same every time
geoips.geoips_utils.get_required_geoips_xarray_attrs()[source]#

Interface deprecated v2.0.

geoips.geoips_utils.is_editable(package_name)[source]#

Return whether or not ‘package_name’ has been installed in editable mode.

Where editable mode is a local package installed via ‘pip install -e <path_to_pkg> and non-editable mode is a local package installed via ‘pip install <pact_to_pkg>.

If the package under package_name doesn’t exist, raise a ValueError reporting that.

Parameters:

package_name (str) –

  • The name of the pip installed local package. (ie. “geoips”, “recenter_tc”, ..)

Returns:

editable –

  • The truth value as to whether or not the package was installed in editable mode.

Return type:

bool

geoips.geoips_utils.load_all_yaml_plugins()[source]#

Find all YAML plugins in registered plugin packages.

Search the plugins directory of each registered plugin package for files ending in .yaml. Read each plugin file

geoips.geoips_utils.log_friendly_subprocess_stdouterr(subprocess_output, std_type)[source]#

Make a log friendly string for either stdout or stderr.

Parameters:

  • subprocess_output (subprocess.run output) – Output of subprocess.run with check_output=True

  • std_type (str) – Specify either stdout or stderr

Returns:

Log friendly string of stdout/stderr

Return type:

str

geoips.geoips_utils.merge_nested_dicts(dest, src, in_place=True, replace=False)[source]#

Perform an in-place merge of src into dest.

Performs an in-place merge of src into dest while preserving any values that already exist in dest.

geoips.geoips_utils.order_paths_from_least_to_most_specific(paths)[source]#

Orders a list of filesystem paths from least to most specific.

This function takes a list of filesystem paths and returns a new list of paths ordered from the least specific (higher-level directories) to the most specific (subdirectories and files). It expands environmental variables in paths.

Parameters:

paths (list of str or pathlib.Path) – A list of filesystem paths to be ordered.

Returns:

A list of filesystem paths ordered from least to most specific.

Return type:

list of pathlib.Path

Examples

>>> paths = [
...     '/home/user/docs/',
...     '/home/user/images/',
...     '/home/user/',
...     '/home/user/images/photo.jpg'
...     '/home/user/docs/report.txt',
... ]
>>> order_paths_from_least_to_most_specific(paths)
[PosixPath('/home/user/'),
 PosixPath('/home/user/docs/'),
 PosixPath('/home/user/images/'),
 PosixPath('/home/user/docs/report.txt'),
 PosixPath('/home/user/images/photo.jpg')]
geoips.geoips_utils.output_process_times(process_datetimes, num_jobs=None, job_str='GeoIPS 2')[source]#

Calculate and print the process times from the process_datetimes dictionary.

Parameters:

process_datetimes (dict) –

dictionary formatted as follows:

  • process_datetimes['overall_start'] - overall start datetime of the entire script

  • process_datetimes['overall_end'] - overall end datetime of the entire script

  • process_datetimes[process_name]['start'] - start time of an individual process

  • process_datetimes[process_name]['end'] - end time of an individual process

geoips.geoips_utils.print_cmd_output(stdout, stderr, output_log_fname=None, stdout_newline_replace_val=None, use_logging=True, use_print=False, prefix=True)[source]#

Print the command output from subprocess.

geoips.geoips_utils.remove_empty_lines(lines, std_type)[source]#

Remove empty lines from stdout and stderr.

geoips.geoips_utils.remove_unsupported_kwargs(module, requested_kwargs)[source]#

Remove unsupported keyword arguments.

geoips.geoips_utils.replace_geoips_paths(path, curly_braces=False)[source]#

Replace specified sub-paths in path with related environment variable names.

This function replaces paths in the provided path with their corresponding environment variable names. This is useful for generating output paths or metadata that are independent of specific installation directories.

For example, it can replace:

  • '/home/user/geoproc/geoips_packages' with '$GEOIPS_PACKAGES_DIR'

  • '/home/user/geoproc/geoips_outdirs' with '$GEOIPS_OUTDIRS'

  • '/home/user/geoproc' with '$GEOIPS_BASEDIR'

Parameters:

  • path (str or pathlib.Path) – The path in which to replace base paths.

  • curly_braces (bool, default=False) – If True, includes curly braces in the environment variables (e.g., '${GEOIPS_BASEDIR}'), otherwise excludes them (e.g., '$GEOIPS_BASEDIR').

Returns:

The path with specified base paths replaced with environment variable names.

Return type:

str

Notes

The function iterates over the provided replace_paths in reverse order (from most specific to least specific) and replaces the first matching base path in the given path with the corresponding environment variable name.

Examples

>>> path = '/home/user/geoproc/geoips_packages/module/file.py'
>>> replace_geoips_paths(path)
'$GEOIPS_PACKAGES_DIR/module/file.py'
geoips.geoips_utils.replace_geoips_paths_in_dict(replace_dict, curly_braces=False)[source]#

Replace geoips paths in every path-based element within a dictionary.

geoips.geoips_utils.replace_geoips_paths_in_list(replace_list, curly_braces=False)[source]#

Replace GeoIPS paths with geoips settings in elements of a list.

This function iterates over each element in the provided replace_list, attempting to replace GeoIPS paths within each element using the replace_geoips_paths function. If an element raises a TypeError when cast to a pathlib path, it is skipped.

Parameters:

  • replace_list (list) – A list of elements to process. Elements can be of any type, but only those that are Path-like will be processed.

  • curly_braces (bool, optional) – Passed to replace_geoips_paths

Returns:

A new list containing the elements with GeoIPS paths replaced where possible. Elements that could not be processed are included unchanged.

Return type:

list

Examples

>>> replace_geoips_paths_in_list(['/home/geoips/data/project',
... 'no_replacement_here'])
['$GEOIPS_DATA_DIR/project', 'no_replacement_here']

See also

replace_geoips_paths

Function used to replace GeoIPS paths in individual elements.

geoips.scripting module#

User-facing helpers for scripting with GeoIPS plugins.

class geoips.scripting.RetentionPolicy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: StrEnum

current_only = 'current_only'#
keep_all = 'keep_all'#
metadata_only = 'metadata_only'#
geoips.scripting.add_data_step(tree, data, *, step_id, retention_policy=None)[source]#

Attach user-created or user-modified data as a manual script step.

Parameters:

  • tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

  • data (xarray.DataTree, DataTreeDitto, xarray.Dataset, xarray.DataArray, or) – convertible object Data to attach as the new manual step.

  • step_id (str) – Step id for the inserted data.

  • retention_policy (RetentionPolicy or str, optional) – Retention policy override for the inserted data step.

Returns:

The same root script DataTree, updated in place and returned for convenient chaining.

Return type:

xarray.DataTree

geoips.scripting.attach_plugin_result(tree, step_data, *, step_id=None, plugin_kind, plugin_name=None, start_time=None, end_time=None, retention_policy=None)[source]#

Attach a scripted plugin result to an initialized script DataTree.

This internal helper records one plugin result under the supplied step_id. If step_id is omitted, the plugin name is used. Duplicate step names are rejected so scripts do not accidentally clobber earlier results.

Parameters:

  • tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

  • step_data (xarray.DataTree, DataTreeDitto, xarray.Dataset, xarray.DataArray, or) – convertible object Result returned by a plugin call. Values must either already be DataTree-compatible or be convertible by DataTreeDitto. Scalars and other simple values should be wrapped in a Dataset, DataArray, or supported container such as a dict before attaching.

  • step_id (str, optional) – Name to use for the attached result node. Defaults to plugin_name.

  • plugin_kind (str) – Registered plugin kind that produced step_data, or "manual" for user-created script data. Values are validated against the cached GeoIPS interface list plus "manual".

  • plugin_name (str, optional) – Plugin name that produced step_data. Required for registered plugin kinds. For plugin_kind="manual", omitted plugin names are recorded as "manual".

  • start_time (datetime.datetime, optional) – Step start datetime. Defaults to the current UTC time for registered plugin kinds. For plugin_kind="manual", omitted times are recorded as None.

  • end_time (datetime.datetime, optional) – Step end datetime. Defaults to the current UTC time for registered plugin kinds. For plugin_kind="manual", omitted times are recorded as None.

  • retention_policy (RetentionPolicy or str, optional) – Retention policy used for this step. Defaults to the policy stored on the root script DataTree.

Returns:

The same root script DataTree, updated in place and returned for convenient chaining.

Return type:

xarray.DataTree

Raises:

  • ValueError – If tree is not an initialized script DataTree, if no step name can be determined, if the step name already exists, or if the retention policy is invalid.

  • TypeError – If step_data cannot be converted to a script DataTree node.

geoips.scripting.get_current_data(tree)[source]#

Return the dataset from the most recent data-containing script step.

Parameters:

tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

Returns:

Dataset from the most recent top-level reader, interpolator, algorithm, or manual step containing data variables, including data stored in child nodes such as reader outputs. The returned object is a mutable, coord-normalized dataset copy intended for inspection, modification, and reinsertion with add_data_step.

Return type:

xarray.Dataset

Raises:

ValueError – If tree is not an initialized script DataTree or no data-containing step is available.

geoips.scripting.get_output_products(tree, step_id=None)[source]#

Return products written by scripted output formatter steps.

Parameters:

  • tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

  • step_id (str, optional) – Specific output formatter step to inspect. If omitted, products from all steps with attrs["output_products"] are returned in tree order.

Returns:

Output products recorded by output formatter steps.

Return type:

list

Raises:

ValueError – If tree is not an initialized script tree, step_id is unknown, or no output products are available.

Examples

>>> from geoips.scripting import get_output_products
>>> output_products = get_output_products(tree)
>>> output_products
["/path/to/output.png"]
geoips.scripting.initialize_script_tree(name, retention_policy, **attrs)[source]#

Initialize a root DataTree for OBP-style scripted plugin calls.

The returned tree is intended to be passed as the data argument to direct plugin calls. Plugins infer script-mode OBP behavior from the tree’s execution_mode metadata and attach results back onto this script tree using step_id.

Parameters:

  • name (str) – Name for the root script DataTree.

  • retention_policy (RetentionPolicy or str) – Retention policy to apply after scripted plugin calls. May be supplied as a RetentionPolicy value or its string value. May be overridden on individual steps.

  • attrs (dict) – Additional root-level metadata to store on the script DataTree.

Returns:

Root DataTree with standard script execution metadata.

Return type:

xarray.DataTree

Raises:

ValueError – If retention_policy is not recognized, or if attrs attempts to override reserved script metadata fields.

Examples

>>> from geoips.scripting import RetentionPolicy, initialize_script_tree
>>> tree = initialize_script_tree(
...     "abi_infrared_test",
...     retention_policy=RetentionPolicy.metadata_only,
... )
>>> tree = reader_plugin(
...     data=tree,
...     filenames=fnames,
...     step_id="read_data",
... )

Module contents#

The Geolocated Information Processing System (GeoIPS).

GeoIPS ® Base Package#

The GeoIPS Base Package provides a Python 3 based architecture supporting a wide variety of satellite and weather data processing. The modular nature of the GeoIPS base infrastructure also allows plug-and-play capability for user-specified custom functionality.

Homepage: NRLMMD-GEOIPS/geoips