geoips.utils.types package

Submodules

geoips.utils.types.converter_registry module

Generalised bidirectional type-converter registry.

TypeConverterRegistry stores one-way converters keyed by (source_type, target_type) tuples and dispatches with priority-matching (exact type, then isinstance MRO-ordered).

It is independent of DataTreeDitto so that both the tree-storage layer and the plugin-lifecycle hooks can share the same converters.

class geoips.utils.types.converter_registry.TypeConverterRegistry

Bases: object

Bidirectional type converter registry with priority dispatch.

Stores one-way (source → target) converters and dispatches calls using exact-type matching first, falling back to isinstance-based matching with MRO ordering (most specific registered type wins).

A module-level singleton converter_registry is created at import time for use throughout the framework.

Examples

>>> registry = TypeConverterRegistry()
>>> registry.register(np.ndarray, xr.Dataset, numpy_to_dataset)
>>> ds = registry.convert(np.array([1, 2, 3]), xr.Dataset)
>>> isinstance(ds, xr.Dataset)
True

can_convert(obj: Any, target: type) → bool

Return True if a conversion path exists for obj → target.

convert(obj: Any, target: type, **kwargs: Any) → Any

Convert obj to target using a registered converter.

Matching order:

1. Exact type(obj) → target entry.

2. isinstance-based match, ordered by the object’s Method Resolution Order (MRO) so the most specific registered base class wins. The MRO is the linearized order Python searches an object’s class hierarchy (type(obj).__mro__); see https://docs.python.org/3/howto/mro.html. For example, a MaskedArray (a subclass of ndarray) prefers a registered MaskedArray converter over an ndarray one.

Parameters:

• obj (Any) – Object to convert.

• target (type) – Desired target type.

• kwargs – Forwarded to the converter function.

Returns:

Converted object (guaranteed to be an instance of target when a converter is found).

Return type:

Any

Raises:

TypeError – If no converter is registered for the requested path.

register(source: type, target: type, converter: Any) → None

Register a one-way converter from source to target.

Parameters:

• source (type) – Source type.

• target (type) – Target type.

• converter (callable) – converter(obj, **kwargs) must return an instance of target.

register_bidirectional(type_a: type, type_b: type, a_to_b: Any, b_to_a: Any) → None

Register a pair of converters for round-trip conversion.

Convenience wrapper around two register() calls.

Parameters:

• type_a (type) – First type.

• type_b (type) – Second type.

• a_to_b (callable) – Convert type_a → type_b.

• b_to_a (callable) – Convert type_b → type_a.

property registered_types: dict[type, set[type]]

{target_type, …}} summary.

Type:

Return {source_type

geoips.utils.types.converters module

Pure type-conversion functions and FamilyConversionSpec.

Every converter here is a standalone function with no side effects, independently testable, and reusable by DataTreeDitto, the TypeConverterRegistry, and the plugin-lifecycle hooks.

Metadata policy

• _ditto_* keys in attrs — storage-layer metadata used by DataTreeDitto for round-trip recovery (original type, shape, dtype, …).

• _conv_* keys in attrs — semantic-conversion metadata used by the plugin-lifecycle hooks to record context (e.g. variable ordering, mask presence).

class geoips.utils.types.converters.FamilyConversionSpec(input_type: type | None = None, input_converter: Callable | None = None, output_type: type | None = None, output_converter: Callable | None = None)

Bases: object

Defines the type conversions needed for a single plugin family.

Parameters:

• input_type (type or None) – The type expected by the plugin’s call() after the input converter has been applied. None means “no conversion”.

• input_converter (callable or None) – converter(dataset) → input_type. Called by _pre_call.

• output_type (type or None) – The type produced by the plugin’s call().

• output_converter (callable or None) – converter(plugin_result) → xr.Dataset. Called by _post_call. None means the plugin already returns a Dataset.

input_converter: Callable | None = None

input_type: type | None = None

output_converter: Callable | None = None

output_type: type | None = None

geoips.utils.types.converters.dataarray_to_dataset(obj: DataArray, **kwargs: Any) → Dataset

Convert a DataArray to a Dataset for DataTreeDitto storage.

Parameters:

• obj (xr.DataArray) –

• kwargs (Ignored.) –

Return type:

xr.Dataset

geoips.utils.types.converters.dataset_dict_to_dataset(dct: dict[str, xarray.core.dataset.Dataset], **kwargs: Any) → Dataset

Merge a dict of datasets into a single xr.Dataset.

Uses xarray.merge for alignment. The dict insertion order is preserved via a _conv_var_order attribute.

Parameters:

• dct (dict[str, xr.Dataset]) –

• kwargs (Ignored.) –

Return type:

xr.Dataset

geoips.utils.types.converters.dataset_to_dataarray(dataset: Dataset, **kwargs: Any) → DataArray

Recover an xr.DataArray from a DataArray-origin xr.Dataset.

Parameters:

• dataset (xr.Dataset) –

• kwargs (Ignored.) –

Return type:

xr.DataArray

geoips.utils.types.converters.dataset_to_dataset_dict(dataset: Dataset, **kwargs: Any) → dict[str, xarray.core.dataset.Dataset]

Split a multi-variable Dataset into a dict of single-variable datasets.

Keys are the data-variable names; each value is a Dataset containing exactly one data variable (plus shared coordinates).

Parameters:

• dataset (xr.Dataset) –

• kwargs (Ignored.) –

Return type:

dict[str, xr.Dataset]

geoips.utils.types.converters.dataset_to_dict(dataset: Dataset, **kwargs: Any) → dict

Recover a plain dict from a dict-origin xr.Dataset.

Walks _ditto_keys to preserve original insertion order.

Parameters:

• dataset (xr.Dataset) –

• kwargs (Ignored.) –

Return type:

dict

geoips.utils.types.converters.dataset_to_list(dataset: Dataset, **kwargs: Any) → list

Recover a list from attrs.

Parameters:

• dataset (xr.Dataset) –

• kwargs (Ignored.) –

Returns:

The stored list value.

Return type:

list

geoips.utils.types.converters.dataset_to_masked_array(dataset: Dataset, **kwargs: Any) → MaskedArray

Reconstruct a numpy.ma.MaskedArray from a dataset.

Uses _conv_mask_var and _conv_fill_value metadata written by masked_array_to_dataset.

Parameters:

• dataset (xr.Dataset) –

• kwargs (Ignored.) –

Return type:

np.ma.MaskedArray

geoips.utils.types.converters.dataset_to_numpy(dataset: Dataset, **kwargs: Any) → ndarray

Extract a numpy array from a dataset produced by numpy_to_dataset.

Uses _ditto_var_name metadata to locate the correct data variable; falls back to the first data variable.

Parameters:

• dataset (xr.Dataset) –

• kwargs (Ignored.) –

Return type:

np.ndarray

geoips.utils.types.converters.dataset_vars_to_numpy_list(dataset: Dataset, **kwargs: Any) → list[numpy.ndarray]

Extract every data variable as a list of arrays.

Arrays are returned in insertion order (the order they appear when iterating dataset.data_vars). This is stable for a given dataset but callers that need a specific ordering should sort or filter the list themselves.

Parameters:

• dataset (xr.Dataset) –

• kwargs (Ignored — future slot for variables subset / ordering context.) –

Returns:

One array per data variable, in insertion order.

Return type:

list[np.ndarray]

geoips.utils.types.converters.dict_to_dataset(obj: dict, **kwargs: Any) → Dataset

Convert a plain dict to an xr.Dataset for DataTreeDitto storage.

Numpy-array values become data variables (each with synthetic dim names of the form <key>_dim_<i>); scalar values (int, float, str, bool, None) become attrs under a _ditto_attr_<key> namespace; any other value type raises TypeError. Original insertion order is preserved in _ditto_keys.

Parameters:

• obj (dict Dictionary of numpy arrays and/or scalars.) –

• kwargs (Ignored.) –

Return type:

xr.Dataset

geoips.utils.types.converters.list_to_dataset(obj: list, **kwargs: Any) → Dataset

Store a plain list as attrs in a Dataset.

Parameters:

• obj (list List of any JSON-serializable values (e.g. filenames).) –

• kwargs (Ignored.) –

Returns:

Dataset with _ditto_list_value in attrs.

Return type:

xr.Dataset

geoips.utils.types.converters.masked_array_to_dataset(obj: MaskedArray, name: str = 'data', dims: list[str] | None = None, **kwargs: Any) → Dataset

Convert a numpy.ma.MaskedArray to xr.Dataset.

Stores the mask as a companion data variable ({name}_mask) and the fill_value in attrs so that dataset_to_masked_array can reconstruct the original MaskedArray faithfully.

Parameters:

• obj (np.ma.MaskedArray) –

• name (str Base data-variable name.) –

• dims (list[str] or None) –

• kwargs (Ignored.) –

Return type:

xr.Dataset

geoips.utils.types.converters.numpy_to_dataset(obj: ndarray, name: str = 'data', dims: list[str] | None = None, **kwargs: Any) → Dataset

Convert a numpy array to an xr.Dataset with _ditto_* metadata.

Parameters:

• obj (np.ndarray) – Input array (may be a regular ndarray or MaskedArray).

• name (str) – Data-variable name in the resulting dataset.

• dims (list[str] or None) – Dimension names. Auto-generated (dim_0, dim_1, …) when None.

• kwargs (Ignored — forward-compatibility slot.) –

Return type:

xr.Dataset

geoips.utils.types.datatree_converters module

Converter registration for the shared type-converter registry.

Registers converters for dict, xr.DataArray, np.ndarray, np.ma.MaskedArray, and list on the shared TypeConverterRegistry singleton – the live dispatch path used by the plugin-lifecycle hooks and by DataTreeDitto conversions.

This module contains only wiring – every converter function lives in the single canonical module geoips.utils.types.converters. This module imports those functions and registers them.

Registration is idempotent – calling this module twice (e.g. via reload) will not produce duplicate entries.

geoips.utils.types.datatree_ditto module

DataTreeDitto: a DataTree that auto-converts non-xarray payloads.

Extends xarray.DataTree so that non-xarray objects (numpy arrays, dicts, etc.) assigned into the tree are converted to Dataset objects on the way in and recovered to their original type on the way out, using the shared converter_registry.

class geoips.utils.types.datatree_ditto.DataTreeDitto(dataset=None, children=None, name=None)

Bases: DataTree

A DataTree subclass that automatically converts non-xarray objects to datasets.

DataTreeDitto extends xarray’s DataTree to automatically handle conversion of various data types (numpy arrays, etc.) to xarray Datasets while preserving metadata needed for round-trip conversion back to original types.

Pokemon, the world’s most valuable intellectual property, has a pokemon named “ditto” that duplicates its opponents behavior and structure with only slight difference. This class - DataTreeDitto - behaves like to xarray DataTrees with slight differences to accommodate a wider variety of data types.

Examples

>>> import numpy as np
>>> arr = np.array([[1, 2], [3, 4]])
>>> dt = DataTreeDitto(arr)
>>> dt.ds.data.values.tolist()
[[1, 2], [3, 4]]
>>> dt.get_original().tolist()
[[1, 2], [3, 4]]

filter(filterfunc: Callable[[DataTree], bool]) → DataTree

Filter nodes according to a specified condition.

Returns a new tree containing only the nodes in the original tree for which fitlerfunc(node) is True. Will also contain empty nodes at intermediate positions if required to support leaves.

Parameters:

filterfunc (function) – A function which accepts only one DataTree - the node on which filterfunc will be called.

Return type:

DataTree

See also

match, pipe, map_over_datasets

classmethod from_datatree(dt: DataTree) → DataTreeDitto

Convert a plain DataTree to DataTreeDitto recursively.

Public API for converting standard DataTree instances returned by xarray operations back to DataTreeDitto.

Parameters:

dt (DataTree) – DataTree instance to convert.

Returns:

Converted DataTreeDitto with all children also converted.

Return type:

DataTreeDitto

get(key, default=None)

Mapping-style get.

Dict-origin nodes return the wrapped dict’s value (or default); all other nodes delegate to DataTree.get. This deliberately does not route through __getitem__ so it stays safe for xarray’s internal tree traversal (which calls node.get(part)).

get_original(path: str = '.') → Any

Get the original object (before conversion) at the specified path.

Parameters:

path (str, default ".") – Path to the node. Use “.” for current node, or child names separated by “/” for nested nodes.

Returns:

Original object if conversion metadata exists, otherwise the dataset. Returns None if the node contains no data.

Return type:

Any

Notes

Performance and memory behavior depend on the registered converter: some converters duplicate data while others reuse the underlying array via views. The returned object shares shallow-copy semantics with the stored data — modifying mutable elements of the returned object may affect the data inside the tree.

Examples

>>> arr = np.array([1, 2, 3])
>>> dt = DataTreeDitto(arr)
>>> original = dt.get_original()
>>> original.tolist()
[1, 2, 3]
>>> dt["child"] = np.array([4, 5, 6])
>>> dt.get_original("child").tolist()
[4, 5, 6]

items()

Return wrapped-dict items for dict-origin nodes, else child items.

keys()

Return wrapped-dict keys for dict-origin nodes, else child names.

list_converted_nodes() → dict[str, str]

List all nodes that contain converted objects and their original types.

Returns:

Mapping of node paths to original object type names. Paths use “/” as separator, with “/” representing the root node.

Return type:

dict of str to str

Examples

>>> dt = DataTreeDitto(np.array([1, 2, 3]))
>>> dt["child"] = np.array([4, 5, 6])
>>> converted = dt.list_converted_nodes()
>>> "/" in converted
True
>>> "child" in converted
True
>>> converted["/"]
'numpy.ndarray'

map_over_datasets(func: Callable[[...], Any], *args: Any, kwargs: Mapping[str, Any] | None = None) → DataTree | tuple[xarray.core.datatree.DataTree, ...]

Map a function over datasets, returning DataTreeDitto output.

match(pattern: str) → DataTree

Return nodes with paths matching pattern.

Uses unix glob-like syntax for pattern-matching.

Parameters:

pattern (str) – A pattern to match each node path against.

Return type:

DataTree

See also

filter, pipe, map_over_datasets

Examples

>>> dt = DataTree.from_dict(
...     {
...         "/a/A": None,
...         "/a/B": None,
...         "/b/A": None,
...         "/b/B": None,
...     }
... )
>>> dt.match("*/B")
<xarray.DataTree>
Group: /
├── Group: /a
│   └── Group: /a/B
└── Group: /b
    └── Group: /b/B

mean(dim=None, *, skipna=None, keep_attrs=None, **kwargs)

Reduce this tree by mean, returning DataTreeDitto output.

to_datatree() → DataTree

Convert to a standard DataTree with all nodes as xarray objects.

Returns:

A standard DataTree instance with all converted objects as datasets. Conversion metadata is preserved in dataset attributes.

Return type:

DataTree

Examples

>>> arr = np.array([1, 2, 3])
>>> dt = DataTreeDitto(arr, name="test")
>>> standard_dt = dt.to_datatree()
>>> isinstance(standard_dt, DataTree)
True
>>> standard_dt.ds.data.values.tolist()
[1, 2, 3]

values()

Return wrapped-dict values for dict-origin nodes, else children.

geoips.utils.types.datatree_helpers module

Neutral helpers for flattening xr.DataTree nodes into mutable Datasets.

These helpers previously lived as duplicated methods on plugin base classes (BaseClassPlugin._to_mutable_dataset) and inside geoips.utils.types.script_datatree. They have one home now so the interface-level plugin classes (algorithms, coverage_checkers, filename_formatters) can share a single implementation without reaching back into the base plugin class or the scripting module.

The canonical flattener is recursive and runs normalize_geoips_dataset_coords on the result, matching the behavior the legacy plugin _pre_call overrides relied on before this refactor.

geoips.utils.types.datatree_helpers.has_data_vars(node)

Return whether node or any descendant contains data variables.

Parameters:

node (xarray.DataTree) – DataTree node to inspect.

Returns:

True if node itself carries data variables, or any of its descendant nodes do.

Return type:

bool

geoips.utils.types.datatree_helpers.to_mutable_dataset(node)

Return a mutable xr.Dataset flattening node and its descendants.

DataTree.ds returns an immutable DatasetView. Plugins that write back into the dataset (e.g. xobj[product_name] = ...) need a mutable xr.Dataset. This helper recursively collects every descendant that carries data variables, merges them into a single mutable Dataset, propagates node attributes, and applies normalize_geoips_dataset_coords so the result matches the GeoIPS coordinate convention legacy call() methods expect.

Parameters:

node (xarray.DataTree) – DataTree node to flatten. May be a reader-style multi-level tree (root attrs-only, sub-children holding each variable group) or a single-level multi-child tree.

Returns:

A mutable, coord-normalized Dataset merging all data-bearing descendants of node. attrs are merged with node attrs taking precedence.

Return type:

xarray.Dataset

geoips.utils.types.family_conversions module

Family-to-converter mappings for the plugin-lifecycle hooks.

Note

These mappings exist purely for backwards compatibility. They let the DataTree-based order-based procflow drive legacy family-based plugins by converting the shared DataTree payload into the concrete type each legacy family expects (and back again). They are keyed by family only and do not describe per-plugin argument/keyword-argument signatures — those legacy call-signature bindings live in geoips.utils.types.obp_conduits.

New plugins are expected to be DataTree-native (declare data_tree = True) and therefore need no entry here. As legacy families are migrated, their entries should be removed; this module is expected to shrink over time and is a candidate for eventual deletion.

Each mapping entry is a FamilyConversionSpec that tells the _pre_call / _post_call hooks what type the plugin’s family expects as input, what type it returns, and which converter functions to use for the forward and reverse transformations.

These mappings are imported by the interface-level base classes (BaseAlgorithmPlugin, BaseOutputFormatterPlugin, …) via their _family_conversion_map class attribute.

geoips.utils.types.obp_conduits module

The Order-Based Procflow (OBP) conduit binding registry.

OBP_CONDUITS maps an upstream node’s plugin kind to the keyword-argument name and extractor a downstream plugin expects. This previously lived inside class_based_plugin (as _OBP_CONDUITS) and was reached back into by YamlPluginCallable; it now has one home, so new plugin kinds are wired in exactly one place.

This registry holds the bespoke, per-kind call-signature bindings for legacy plugins — i.e. the specific keyword-argument name each legacy plugin family expects and how to pull that value out of an upstream DataTree node. It is the companion to geoips.utils.types.family_conversions, which handles type conversions; this module handles argument wiring. Together they let the DataTree-based OBP drive legacy plugins.

Like family_conversions, these bindings are a backwards-compatibility layer: DataTree-native plugins (data_tree = True) consume the DataTree directly and need no conduit entry, so this registry is expected to shrink as plugins are migrated. (Originally adapted from @srikanth-kumar’s work.)

geoips.utils.types.obp_conduits.OBP_CONDUITS: dict[str, dict] = {'algorithm': {'extract': <function _extract_ds>, 'kwarg': 'xarray_obj'}, 'colormapper': {'extract': <function <lambda>>, 'kwarg': 'mpl_colors_info'}, 'coverage_checker': {'extract': <function <lambda>>, 'kwarg': 'coverage'}, 'feature_annotator': {'extract': <function _extract_annotator_spec>, 'kwarg': 'feature_annotator'}, 'filename_formatter': {'extract': <function <lambda>>, 'kwarg': 'output_filenames'}, 'gridline_annotator': {'extract': <function _extract_annotator_spec>, 'kwarg': 'gridline_annotator'}, 'interpolator': {'extract': <function _extract_ds>, 'kwarg': 'xarray_obj'}, 'manual': {'extract': <function _extract_ds>, 'kwarg': 'xarray_obj'}, 'output_formatter': {'extract': <function <lambda>>, 'kwarg': 'output_products'}, 'product': {'extract': <function _extract_product_name>, 'kwarg': 'product_name'}, 'product_default': {'extract': <function _extract_attrs_dict>, 'kwarg': 'product_default_info'}, 'reader': {'extract': <function _extract_ds>, 'kwarg': 'xarray_obj'}, 'sector': {'extract': <function <lambda>>, 'kwarg': 'area_def'}}

Maps an upstream node’s plugin kind to the downstream kwarg name it feeds and the extractor that pulls the value out of the node. Add a new plugin kind’s wiring here (single location).

geoips.utils.types.obp_conduits.kwarg_name_for_kind(kind: str) → str

Return the downstream kwarg name an upstream kind feeds.

Falls back to kind itself when the kind has no registered conduit.

geoips.utils.types.partial_lexeme module

Lightweight helper for treating singular and plural spellings as equivalent.

A drop‑in str subclass that normalizes English nouns so that their singular and plural forms compare as equal, hash to the same key, and work inside Pydantic v2 models. Ignores capitalization.

Examples

>>> from partial_lexeme import Lexeme
>>> Lexeme("reader") == "readers" == Lexeme("readers")
True
>>> {Lexeme("analyses"): 1} == {"analysis": 1}
True

>>> from pydantic import BaseModel
>>> class _Plugin(BaseModel):
>>>     kind: Lexeme
>>>     interface: Lexeme
>>> model = _Plugin(kind="reader", interface="readers")
>>> assert model.kind == model.interface == "reader"

class geoips.utils.types.partial_lexeme.Lexeme(value: Any)

Bases: str

A string that treats singular and plural spellings as equal.

It behaves exactly like a built‑in str but overrides equality and hashing to use the canonical singular form. This makes singular and plural spellings interchangeable as dict keys, set members, CLI options, etc.

property plural: Lexeme

Return a plural spelling.

property singular: Lexeme

Return the singular spelling.

geoips.utils.types.script_datatree module

Helpers for OBP-style plugin calls from scripts.

class geoips.utils.types.script_datatree.RetentionPolicy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: StrEnum

current_only = 'current_only'

keep_all = 'keep_all'

metadata_only = 'metadata_only'

geoips.utils.types.script_datatree.add_data_step(tree, data, *, step_id, retention_policy=None)

Attach user-created or user-modified data as a manual script step.

Parameters:

• tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

• data (xarray.DataTree, DataTreeDitto, xarray.Dataset, xarray.DataArray, or) – convertible object Data to attach as the new manual step.

• step_id (str) – Step id for the inserted data.

• retention_policy (RetentionPolicy or str, optional) – Retention policy override for the inserted data step.

Returns:

The same root script DataTree, updated in place and returned for convenient chaining.

Return type:

xarray.DataTree

geoips.utils.types.script_datatree.apply_script_retention(tree, current_step_id, retention_policy=None)

Apply a retention policy to an initialized script DataTree.

Parameters:

• tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

• current_step_id (str) – Step id for the current result. This step remains intact for all retention policies.

• retention_policy (RetentionPolicy or str, optional) – Retention policy to apply. Defaults to the root script DataTree policy.

Returns:

The same root script DataTree, updated in place and returned for convenient chaining.

Return type:

xarray.DataTree

Raises:

ValueError – If tree is not an initialized script DataTree, if current_step_id does not exist, or if the retention policy is invalid.

geoips.utils.types.script_datatree.attach_plugin_result(tree, step_data, *, step_id=None, plugin_kind, plugin_name=None, start_time=None, end_time=None, retention_policy=None)

Attach a scripted plugin result to an initialized script DataTree.

This internal helper records one plugin result under the supplied step_id. If step_id is omitted, the plugin name is used. Duplicate step names are rejected so scripts do not accidentally clobber earlier results.

Parameters:

• tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

• step_data (xarray.DataTree, DataTreeDitto, xarray.Dataset, xarray.DataArray, or) – convertible object Result returned by a plugin call. Values must either already be DataTree-compatible or be convertible by DataTreeDitto. Scalars and other simple values should be wrapped in a Dataset, DataArray, or supported container such as a dict before attaching.

• step_id (str, optional) – Name to use for the attached result node. Defaults to plugin_name.

• plugin_kind (str) – Registered plugin kind that produced step_data, or "manual" for user-created script data. Values are validated against the cached GeoIPS interface list plus "manual".

• plugin_name (str, optional) – Plugin name that produced step_data. Required for registered plugin kinds. For plugin_kind="manual", omitted plugin names are recorded as "manual".

• start_time (datetime.datetime, optional) – Step start datetime. Defaults to the current UTC time for registered plugin kinds. For plugin_kind="manual", omitted times are recorded as None.

• end_time (datetime.datetime, optional) – Step end datetime. Defaults to the current UTC time for registered plugin kinds. For plugin_kind="manual", omitted times are recorded as None.

• retention_policy (RetentionPolicy or str, optional) – Retention policy used for this step. Defaults to the policy stored on the root script DataTree.

Returns:

The same root script DataTree, updated in place and returned for convenient chaining.

Return type:

xarray.DataTree

Raises:

• ValueError – If tree is not an initialized script DataTree, if no step name can be determined, if the step name already exists, or if the retention policy is invalid.

• TypeError – If step_data cannot be converted to a script DataTree node.

geoips.utils.types.script_datatree.attach_script_result(script_tree, result, *, plugin, step_id=None, step_start_time=None, retention_policy=None)

Attach a scripted plugin result and return the updated script tree.

Thin convenience wrapper around attach_plugin_result that derives plugin_kind from plugin’s interface and plugin_name from plugin.name, and defaults the end time to the current UTC time. Used by BaseClassPlugin._invoke so the two script-mode attach sites do not duplicate the metadata-derivation logic.

geoips.utils.types.script_datatree.get_current_data(tree)

Return the dataset from the most recent data-containing script step.

Parameters:

tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

Returns:

Dataset from the most recent top-level reader, interpolator, algorithm, or manual step containing data variables, including data stored in child nodes such as reader outputs. The returned object is a mutable, coord-normalized dataset copy intended for inspection, modification, and reinsertion with add_data_step.

Return type:

xarray.Dataset

Raises:

ValueError – If tree is not an initialized script DataTree or no data-containing step is available.

geoips.utils.types.script_datatree.get_output_products(tree, step_id=None)

Return products written by scripted output formatter steps.

Parameters:

• tree (xarray.DataTree) – Root script DataTree created by initialize_script_tree.

• step_id (str, optional) – Specific output formatter step to inspect. If omitted, products from all steps with attrs["output_products"] are returned in tree order.

Returns:

Output products recorded by output formatter steps.

Return type:

list

Raises:

ValueError – If tree is not an initialized script tree, step_id is unknown, or no output products are available.

Examples

>>> from geoips.scripting import get_output_products
>>> output_products = get_output_products(tree)
>>> output_products
["/path/to/output.png"]

geoips.utils.types.script_datatree.initialize_script_tree(name, retention_policy, **attrs)

Initialize a root DataTree for OBP-style scripted plugin calls.

The returned tree is intended to be passed as the data argument to direct plugin calls. Plugins infer script-mode OBP behavior from the tree’s execution_mode metadata and attach results back onto this script tree using step_id.

Parameters:

• name (str) – Name for the root script DataTree.

• retention_policy (RetentionPolicy or str) – Retention policy to apply after scripted plugin calls. May be supplied as a RetentionPolicy value or its string value. May be overridden on individual steps.

• attrs (dict) – Additional root-level metadata to store on the script DataTree.

Returns:

Root DataTree with standard script execution metadata.

Return type:

xarray.DataTree

Raises:

ValueError – If retention_policy is not recognized, or if attrs attempts to override reserved script metadata fields.

Examples

>>> from geoips.scripting import RetentionPolicy, initialize_script_tree
>>> tree = initialize_script_tree(
...     "abi_infrared_test",
...     retention_policy=RetentionPolicy.metadata_only,
... )
>>> tree = reader_plugin(
...     data=tree,
...     filenames=fnames,
...     step_id="read_data",
... )

geoips.utils.types.script_datatree.is_script_tree(tree)

Return whether tree is an initialized script DataTree.

geoips.utils.types.script_datatree.normalize_retention_policy(retention_policy)

Return a canonical retention policy from a string or RetentionPolicy.

This internal helper lets user-facing functions accept either RetentionPolicy.metadata_only style values or equivalent string values such as "metadata_only".

geoips.utils.types.script_datatree.plugin_kind(plugin)

Return the singular plugin kind for script step metadata.

Uses the plugin’s interface attribute (e.g. "filename_formatters") and returns its singular form (e.g. "filename_formatter").

geoips.utils.types.script_datatree.script_call_data(data, kwargs)

Return the data object to pass through plugin preprocessing.

Script calls keep the accumulated script tree in data; the plugin’s own call should operate on the current data input (xarray_obj kwarg when a conduit injected one, otherwise the tree itself).

geoips.utils.types.script_datatree.validate_retention_policy(retention_policy)

Validate a script DataTree retention policy.

geoips.utils.types.tokenization module

Stable tokenization helpers for DataTree provenance.

Wraps dask.base.tokenize to produce deterministic, content-addressed tokens for step outputs, argument dictionaries, and step provenance chains. Tokens are prefixed with "dask:" so future migrations to a different hash provider remain detectable and safe to compare by full-string equality.

geoips.utils.types.tokenization.compute_arguments_hash(arguments: dict[str, Any]) → str

Tokenize a step’s arguments dictionary.

The hash covers keys and values in a stable order so that semantically identical argument dicts produce identical tokens even when supplied via different code paths.

Parameters:

arguments (dict) – The step’s arguments dictionary from the workflow spec.

Returns:

A "dask:"-prefixed token string.

Return type:

str

geoips.utils.types.tokenization.compute_step_output_token(step_output: Any, plugin_name: str, plugin_kind: str, arguments: dict[str, Any], upstream_tokens: dict[str, str] | None = None) → str

Compute a step’s output_token for provenance attrs.

The token MUST change when any of plugin identity, arguments, or upstream tokens change. It MUST NOT change for cosmetic refactors that produce the same data.

Parameters:

• step_output (Any) – The return value of a step’s call() method (a Dataset, DataTree, numpy array, dict, etc.).

• plugin_name (str) – Plugin name (e.g. "abi_netcdf").

• plugin_kind (str) – Plugin kind (e.g. "reader").

• arguments (dict) – The step’s arguments dictionary.

• upstream_tokens (dict[str, str] or None) – Mapping of upstream step-id → output_token, or None for reader steps with no dependencies.

Returns:

A "dask:"-prefixed token, or "untokenizable:<exc>" if tokenization raises an exception.

Return type:

str

geoips.utils.types.tokenization.compute_token(*objs: Any) → str

Compute a stable, deterministic token over the given objects.

Wraps dask.base.tokenize and prefixes its output so that future migrations to a different hash provider remain forward-compatible in stored attrs.

Parameters:

*objs (Any) – Objects to include in the token. The call is positional so that callers cannot accidentally pass the same objects in a different order and get the same token.

Returns:

A "dask:<hex>" token string.

Return type:

str

Examples

>>> compute_token("hello", 42)
'dask:...'
>>> compute_token("hello", 42) == compute_token("hello", 42)
True

geoips.utils.types.yaml_plugin_callable module

Adapter wrapping any YAML plugin dict into a callable DataTree participant.

Every YAML-defined plugin (feature_annotator, gridline_annotator, product, product_default, sector, …) becomes a first-class DAG step that adds its configuration / metadata to the step DataTree.

The Workflow._resolve_plugin static method automatically wraps dict results (from YAML-based interfaces) in a YamlPluginCallable so that the workflow executor can call them uniformly.

class geoips.utils.types.yaml_plugin_callable.YamlPluginCallable(yaml_plugin: dict[str, Any])

Bases: object

Wraps a YAML plugin dict into a callable DataTree participant.

The adapter satisfies the informal protocol that Workflow.call() expects:

plg(data=upstream, **arguments) -> xr.DataTree

It stores the YAML plugin’s spec in the returned DataTree’s ds.attrs together with metadata keys (plugin_kind, output_key) that downstream hooks use to map the child node to the correct keyword argument.

Parameters:

yaml_plugin (dict) – The raw YAML plugin dict as returned by a YAML-based interface’s get_plugin(name). Must contain at least "interface" and "spec"; "name" and "family" are optional.

call(data: DataTree | None = None, **kwargs: Any) → DataTree

Produce a DataTree node carrying the YAML plugin’s specification.

Parameters:

• data (xr.DataTree or None) – Upstream DataTree (ignored for most YAML plugins; gridline annotators may use it for automatic spacing computation).

• kwargs – Step arguments from the workflow specification. Currently forwarded for interface-specific post-processing.

Returns:

A DataTreeDitto whose ds.attrs contain the plugin’s spec dictionary and the standard routing metadata.

Return type:

xr.DataTree

property spec: dict[str, Any]

The spec section of the wrapped YAML plugin dict.

Module contents

Types subpackage init file.

The imports below trigger registration of the shared TypeConverterRegistry and all built-in / extended converters at import time so that both DataTreeDitto and the plugin-lifecycle hooks can discover them.

Note

In a future version the import-time side-effects should be replaced with explicit register_converters() calls from the framework bootstrap entry point.