geoips.plugins.modules.procflows package#

Submodules#

geoips.plugins.modules.procflows.config_based module#

Processing workflow for config-based processing.

geoips.plugins.modules.procflows.config_based.call(fnames, command_line_args=None)[source]#

Workflow for efficiently running all required outputs.

Includes all sectors and products specified in a YAML output config file. Specified via a YAML config file

Parameters:

  • fnames (list) – List of strings specifying full paths to input file names to process

  • command_line_args (dict) – dictionary of command line arguments

Returns:

0 for successful completion, non-zero for error (incorrect comparison, or failed run)

Return type:

int

geoips.plugins.modules.procflows.config_based.get_area_def_list_from_dict(area_defs)[source]#

Get a list of actual area_defs from full dictionary.

Dict returned from get_area_defs_from_available_sectors

geoips.plugins.modules.procflows.config_based.get_area_defs_from_available_sectors(available_sectors_dict, command_line_args, xobjs, variables)[source]#

Get all required area_defs for the given set of parameters.

YAML config parameters (config_dict), command_line_args, xobjs, and required variables. Command line args override config specifications.

Parameters:

  • available_sectors_dict (dict) – Dictionary of all requested sector_types (specified in YAML config)

  • command_line_args (dict) – Dictionary of command line arguments - any command line argument that is also a key in available_sectors_dict[<sector_type>] will replace the value in the available_sectors_dict[<sector_type>]

  • xobjs (dict) – Dictionary of xarray datasets, used in determining start/end time of data files for identifying dynamic sectors

  • variables (list) – List of required variables, for determining center coverage for TCs

Returns:

Dictionary of required area_defs, with area_def.area_id as the dictionary keys. Based on YAML config-specified available_sectors, and command line args

Return type:

dict

Notes

  • Each area_def.area_id key has one or more “sector_types” associated with it.

  • Each sector_type dictionary contains the actual “requested_sector_dict” from the YAML config, and the actual AreaDefinition object that was returned.

    • area_defs[area_def.area_id][sector_type]['requested_sector_dict']

    • area_defs[area_def.area_id][sector_type]['area_def']

geoips.plugins.modules.procflows.config_based.get_bg_xarray(sect_xarrays, area_def, prod_plugin, resampled_read=False, window_start_time=None, window_end_time=None)[source]#

Get background xarray.

Parameters:

  • sect_xarrays (dict of xarray.Dataset) – dictionary of xarray Datasets to pull appropriate background xarray from. This may include multiple products / variables.

  • area_def (pyresample.AreaDefinition) – Spatial region required in the final xarray Datasets.

  • prod_plugin (ProductPlugin) – GeoIPS Product Plugin obtained through interfaces.products.get_plugin(“name”).

  • resampled_read (bool, default=False) – Specify whether a resampled read is required, needed for datatypes that will be read within “get_alg_xarray”

  • window_start_time (datetime.datetime, default=None) – If specified, sector temporally between window_start_time and window_end_time. hours_before_sector_time and hours_after_sector_time are ignored if window start/end time are set!

  • window_end_time (datetime.datetime, default=None) – If specified, sector temporally between window_start_time and window_end_time. hours_before_sector_time and hours_after_sector_time are ignored if window start/end time are set!

Returns:

alg_xarray – xarray Dataset containing the data needed to produce the background for overlay imagery.

Return type:

xarray.Dataset

geoips.plugins.modules.procflows.config_based.get_config_dict(config_yaml_file)[source]#

Populate the full config dictionary from a given YAML config file.

Includes both sector and output specifications.

Parameters:

config_yaml_file (str) – Full path to YAML config file, containing sector and output specifications. YAML config files support environment variables in entries flagged with !ENV

Returns:

Return dictionary of both sector and output specifications, as found in config_yaml_file. The output dictionary references the “sector_types” found in the available_sectors dictionary, each output_type requests a specific “sector_type” to be used for processing.

Return type:

dict

geoips.plugins.modules.procflows.config_based.get_required_outputs(config_dict, sector_type)[source]#

Get only the required outputs from the current sector_type.

geoips.plugins.modules.procflows.config_based.get_resampled_read(config_dict, area_defs, area_def_id, sector_type, reader_plugin, reader_kwargs, fnames, variables)[source]#

Return dictionary of xarray datasets for a given area def.

Xarrays resampled to area_def

geoips.plugins.modules.procflows.config_based.get_sectored_read(config_dict, area_defs, area_def_id, sector_type, reader_plugin, reader_kwargs, fnames, variables)[source]#

Return dictionary of xarray datasets for a given area def.

Xarrays sectored to area_def

geoips.plugins.modules.procflows.config_based.get_variables_from_available_outputs_dict(available_outputs_dict, source_name, sector_types=None)[source]#

Get required variables for all outputs for a given “source_name”.

Outputs specified within the YAML config.

Parameters:

  • available_outputs_dict (dict) – Dictionary of all requested output_types (specified in YAML config)

  • source_name (str) – Find all required variables for the passed “source_name”

  • sector_types (list, default=None) – if sector_types list of strings is passed, only include output_types that require one of the passed “sector_types”

Returns:

List of all required variables for all output products for the given source_name

Return type:

list

geoips.plugins.modules.procflows.config_based.initialize_final_products(final_products, cpath)[source]#

Initialize the final_products dictionary with cpath dict key if needed.

Parameters:

  • final_products (dict) – Dictionary of final products, with keys of final required “compare_path” Products with no compare_path specified are stored with the key “no_comparison”

  • cpath (str) – Key to add to final_products dictionary

Returns:

Return final_products dictionary, updated with current “cpath” key: final_products[cpath][‘files’] = <list_of_files_in_given_cpath>

Return type:

dict

geoips.plugins.modules.procflows.config_based.is_required_sector_type(available_outputs_dict, sector_type)[source]#

Check if current sector is required for any outputs.

Check if a given sector_type is required for any currently requested output_types

Parameters:

  • available_outputs_dict (dict) – Dictionary of all requested output_types (specified in YAML config)

  • sector_type (str) – Determine if any output_types require the currently requested “sector_type”

Returns:

  • True if any output_types require the passed “sector_type”

  • False if no output_types require the passed “sector_type”

Return type:

bool

geoips.plugins.modules.procflows.config_based.process_unsectored_data_outputs(final_products, available_outputs_dict, available_sectors_dict, xobjs, variables, command_line_args=None, write_to_product_db=False, config_dict=None)[source]#

Process unsectored data output.

Loop through all possible outputs, identifying output types that require unsectored data output. Produce all required unsectored data output, update final_products dictionary accordingly, and return final_products dictionary with the new unsectored outputs.

Parameters:

  • final_products (dict) – Dictionary of final products, with keys of final required “compare_path” Products with no compare_path specified are stored with the key “no_comparison”

  • available_outputs_dict (dict) – Dictionary of all available output product specifications

  • available_sectors_dict (dict) – Dictionary of available sector types - we are looking for available sectors that contain the “unsectored” keyword.

  • xobjs (dict) – Dictionary of xarray datasets, for use in producing unsectored output formats

  • variables (list) – List of strings of required variables in the given product.

Returns:

Return final_products dictionary, updated with current “cpath” key: final_products[cpath][‘files’] = <list_of_files_in_given_cpath>

Return type:

dict

geoips.plugins.modules.procflows.config_based.requires_bg(available_outputs_dict, sector_type)[source]#

Check if current sector requires background imagery.

Check if a given sector_type is requested for any product_types that also require background imagery.

Parameters:

  • available_outputs_dict (dict) – Dictionary of all requested output_types (specified in YAML config)

  • sector_type (str) – sector_type to determine if any output_types that require background imagery also request the passed sector_type

Returns:

  • True if any output_types that require background imagery require the passed “sector_type”

  • False if no output_types require both background imagery and the passed “sector_type”

Return type:

bool

geoips.plugins.modules.procflows.config_based.set_comparison_path(output_dict, product_name, output_type, command_line_args=None)[source]#

Replace variables specified by <varname> in compare_path.

Parameters:

  • config (dict) – Dictionary of output specifications, containing key “compare_path”

  • product_name (str) – Current requested product name, all instances of <product> in compare_path replaced with product_name argument

  • output_type (str) – Current requested output type, all instances of <output> in compare_path replaced with output argument

Returns:

Return a single string with the fully specified comparison path for current product

Return type:

str

geoips.plugins.modules.procflows.config_based.update_output_dict_from_command_line_args(output_dict, command_line_args=None)[source]#

Update output dict from command line args.

geoips.plugins.modules.procflows.obp_utils module#

Order-Based Procflow (OBP) utility functions.

geoips.plugins.modules.procflows.obp_utils.remove_keys_with_default_value_plugin_provided(workflow_dict: dict) → dict[source]#

Sanitize the workflow by removing keys with value ‘PLUGIN_PROVIDED’.

‘PLUGIN_PROVIDED’ acts as a secondary default value in the OBP. It is used when the same argument across different plugins of the same type has different default values.

This function recursively removes workflow arguments whose value is ‘PLUGIN_PROVIDED’. The plugin call signature would process these arguments as omitted and populates the default value accordingly.

Parameters:

workflow (dict) – The workflow plugin dictionary to process.

Returns:

workflow dictionary with all keys containing the value ‘plugin_provided’ removed.

Return type:

dict

geoips.plugins.modules.procflows.obp_utils.validate_arguments(apiVersion, interface, arguments)[source]#

Load the correct pydantic argument model and validate arguments.

Where the ‘correct’ argument model is based on apiVersion and interface.

Parameters:

  • apiVersion (str) – The apiVersion of the workflow model that contained these arguments.

  • interface (str) – The name of the interface we’ll provide arguments to.

  • arguments (dict[str, Any]) – A dictionary of arguments to validate against a certain model.

Returns:

validated_arguments – A validated representation of the input arguments.

Return type:

dict[str, Any]

geoips.plugins.modules.procflows.order_based module#

Processing workflow for order-based data source processing.

The OrderBased class replaces the legacy module-level call() function. External callers should resolve through the procflows registry: interfaces.procflows.get_plugin("order_based")(workflow_spec, fnames=fnames).

class geoips.plugins.modules.procflows.order_based.OrderBased(module=None)[source]#

Bases: BaseProcflowPlugin

Execute an order-based process workflow (OBP).

Loads a validated workflow specification, builds a Workflow composite instance, and walks through steps in topological order, collecting output into an xr.DataTree.

call(workflow_spec: WorkflowPluginModel | WorkflowSpecModel | dict, filenames: Any, command_line_args: Any = None, **kwargs: Any)[source]#

Run the order-based procflow.

Parameters:

  • workflow_spec (WorkflowPluginModel, WorkflowSpecModel, or dict) – The workflow specification to execute. May be a pre-validated WorkflowSpecModel model, a raw dictionary that will be validated on entry, or a dict that wraps a spec: field.

  • fnames (list[str] or str or None) – Input filename glob or list of filenames for reader steps.

  • command_line_args (Namespace or None) – Parsed CLI arguments, accepted for parity with the CLI entry point. Not currently consumed by the workflow runtime.

  • kwargs (dict) – Additional keyword arguments forwarded to the Workflow.

Returns:

The fully-populated workflow DataTree.

Return type:

xr.DataTree

family = 'order_based'#
interface = 'procflows'#
name = 'order_based'#
geoips.plugins.modules.procflows.order_based.PLUGIN_CLASS#

alias of OrderBased

geoips.plugins.modules.procflows.order_based.call(workflow_spec, fnames, command_line_args=None, **kwargs)[source]#

Module-level entry point for pluginify discovery.

Delegates to OrderBased.call.

geoips.plugins.modules.procflows.single_source module#

Processing workflow for single data source processing.

geoips.plugins.modules.procflows.single_source.add_attrs_from_area_def(final_xarray, source_xarray, area_def)[source]#

Add attributes from an area_def.

geoips.plugins.modules.procflows.single_source.add_filename_extra_field(xarray_obj, field_name, field_value)[source]#

Add filename extra field.

geoips.plugins.modules.procflows.single_source.apply_alg_after_interp(interp_xarray, area_def, alg_plugin, alg_args, prod_plugin, variables, processed_xarrays)[source]#

Apply algorithm after interpolation.

MLS need to add ability here to pull from processed_xarrays if algorithm was already applied.

geoips.plugins.modules.procflows.single_source.apply_alg_first(alg_plugin, alg_args, prod_plugin, curr_sect_xarrays, sect_xarrays, variables, variable_names, area_def)[source]#

Apply algorithm appropriately based on algorithm family.

MLS Inexplicably some of these use curr_sect_xarrays, and some use sect_xarrays. Also, some use variables and some use variable_names. I am guessing there is no reason for the difference, but maintaining the original functionality for now.

geoips.plugins.modules.procflows.single_source.apply_alg_list_numpy_to_numpy(alg_xarray, alg_plugin, alg_args, prod_plugin, sect_xarrays, variables)[source]#

Apply list_numpy_to_numpy algorithm.

geoips.plugins.modules.procflows.single_source.apply_alg_xarray_dict_area_def_to_numpy(alg_xarray, alg_plugin, alg_args, prod_plugin, sect_xarrays, area_def)[source]#

Apply xarray_dict_area_def_to_numpy algorithm.

geoips.plugins.modules.procflows.single_source.apply_alg_xarray_dict_to_xarray(alg_plugin, alg_args, sect_xarrays)[source]#

Apply xarray_dict_to_xarray algorithm.

geoips.plugins.modules.procflows.single_source.apply_alg_xarray_dict_to_xarray_dict(alg_plugin, alg_args, sect_xarrays)[source]#

Apply xarray_dict_to_xarray algorithm.

geoips.plugins.modules.procflows.single_source.apply_alg_xarray_to_numpy(alg_xarray, alg_plugin, alg_args, prod_plugin, sect_xarrays, variable_names)[source]#

Apply xarray_to_numpy algorithm.

geoips.plugins.modules.procflows.single_source.apply_alg_xarray_to_xarray(alg_plugin, alg_args, prod_plugin, sect_xarrays, variable_names)[source]#

Apply xarray_to_xarray algorithm.

geoips.plugins.modules.procflows.single_source.apply_interp_after_alg(alg_xarray, interp_plugin, interp_args, prod_plugin, area_def, processed_xarrays)[source]#

Apply interpolation after algorithm.

geoips.plugins.modules.procflows.single_source.apply_interp_first(variables, curr_sect_xarrays, prod_plugin, datasets_for_vars, resampled_read, area_def, processed_xarrays)[source]#

Apply interpolation first.

For product types that involve interpolation before algorithm.

geoips.plugins.modules.procflows.single_source.call(fnames, command_line_args=None)[source]#

Workflow for running products from a single data source.

Parameters:

  • fnames (list) – List of strings specifying full paths to input file names to process

  • command_line_args (dict) – dictionary of command line arguments

Returns:

Return list of strings specifying full paths to output products that were produced

Return type:

list

See also

geoips.commandline.args

Complete list of available command line args.

geoips.plugins.modules.procflows.single_source.combine_filename_extra_fields(source_xarray, dest_xarray)[source]#

Combine filename extra fields.

geoips.plugins.modules.procflows.single_source.get_alg_and_interp_plugins(prod_plugin)[source]#

Get algorithm and interpolator plugins from prod_plugin definition.

geoips.plugins.modules.procflows.single_source.get_alg_xarray(sect_xarrays, area_def, prod_plugin, processed_xarrays=None, resector=True, resampled_read=False, variable_names=None, window_start_time=None, window_end_time=None, pid_track=None)[source]#

Get alg xarray.

Parameters:

  • sect_xarrays (dict of xarray.Dataset) – dictionary of xarray Datasets to apply algorithm.

  • area_def (pyresample.AreaDefinition) – Spatial region required in the final xarray Datasets.

  • prod_plugin (ProductPlugin) – GeoIPS Product Plugin obtained through interfaces.products.get_plugin(“name”).

  • resector (bool, default=True) – Specify whether to resector the data prior to applying the algorithm.

  • resampled_read (bool, default=False) – Specify whether a resampled read is required, needed for datatypes that will be read within “get_alg_xarray”

  • variable_names (list of str) – List of variable names within xarray Datasets to include in the final sectored xarray Datasets

  • window_start_time (datetime.datetime, default=None) – If specified, sector temporally between window_start_time and window_end_time.

  • window_end_time (datetime.datetime, default=None) – If specified, sector temporally between window_start_time and window_end_time.

Returns:

xarray Dataset containing the final data after interpolation, algorithm, resectoring, etc have been applied.

Return type:

xarray.Dataset

geoips.plugins.modules.procflows.single_source.get_area_defs_from_command_line_args(command_line_args, xobjs, variables=None, filter_time=True)[source]#

Get area def from command line args.

geoips.plugins.modules.procflows.single_source.get_filename(filename_formatter, prod_plugin=None, alg_xarray=None, area_def=None, supported_filenamer_types=None, output_dict=None, filename_formatter_kwargs=None)[source]#

Get filename.

geoips.plugins.modules.procflows.single_source.get_interp_plugin_from_product(prod_plugin)[source]#

Get the interpolator plugin from the product spec.

Reassign interp_plugin based on CURRENT sect_xarray Allow re-defining interpolation for different datasets.

geoips.plugins.modules.procflows.single_source.get_output_filenames(fname_formats, output_dict, prod_plugin, xarray_obj=None, area_def=None, supported_filenamer_types=None)[source]#

Get output filenames.

geoips.plugins.modules.procflows.single_source.get_unique_dataset_key(area_def, xobj)[source]#

Get a unique id for xarray dataset.

geoips.plugins.modules.procflows.single_source.output_all_metadata(output_dict, output_fnames, metadata_fnames, xarray_obj, area_def=None)[source]#

Output all metadata.

geoips.plugins.modules.procflows.single_source.pad_area_definition(area_def, source_name=None, force_pad=False, x_scale_factor=1.5, y_scale_factor=1.5)[source]#

Pad area definition.

geoips.plugins.modules.procflows.single_source.perform_interpolation(interp_plugin, area_def, sect_xarray, interp_xarray, interp_args, processed_xarrays)[source]#

Perform standard interpolation.

geoips.plugins.modules.procflows.single_source.plot_data(output_dict, alg_xarray, area_def, prod_plugin, output_kwargs, fused_xarray_dict=None, no_output=False)[source]#

Plot data.

alg_xarray used for filename formats, etc. If included, fused_xarray_dict used for output format call

geoips.plugins.modules.procflows.single_source.process_sectored_data_output(xobjs, variables, prod_plugin, output_dict, area_def=None, pid_track=None)[source]#

Process sectored data output.

If current product family requires a sectored dictionary of xarrays, does not apply an algorithm, and DOES require an area definition, call ‘process_xarray_dict_to_output_format’, store the result in a list, and return it.

geoips.plugins.modules.procflows.single_source.process_xarray_dict_to_output_format(xobjs, variables, prod_plugin, output_dict, area_def=None, pid_track=None)[source]#

Process xarray dict to output format.

geoips.plugins.modules.procflows.single_source.remove_unsupported_kwargs(module, requested_kwargs)[source]#

Remove unsupported keyword arguments.

geoips.plugins.modules.procflows.single_source.resector_xarrays(resector, sect_xarrays, area_def, variables, window_start_time, window_end_time)[source]#

Resector xarrays if requested.

geoips.plugins.modules.procflows.single_source.select_variables_to_interp(interp_args, area_def, source_xarray, interp_xarray, processed_xarrays)[source]#

Select interpolation variables.

geoips.plugins.modules.procflows.single_source.use_variable_from_current_dataset(varname, key, variables, sect_xarray, interp_xarray, resampled_read, datasets_for_vars)[source]#

Use the variable from the current dataset.

If a specific dataset was requested for the current variable, and this dataset was NOT requested via a resampled_read (in which case the native datasets won’t exist, only the resampled dataset), then use the appropriately requested dataset.

geoips.plugins.modules.procflows.single_source.verify_area_def(area_defs, check_area_def, data_start_datetime, data_end_datetime, time_range_hours=3)[source]#

Verify current area definition is the closest to the actual data time.

When looping through multiple dynamic area definitions for a full data file that temporally covers more than one dynamic area_def, there is no way of knowing which dynamic area_def has the best coverage until AFTER we have actually sectored the data to the specific area_def.

Call this utility on the current area_def (check_area_def) for the sectored data file, plus the full list of area definitions (area_defs) that cover the FULL data file.

Returns:

  • True if the current area definition is NOT dynamic

  • True if the current area definition IS dynamic and is the closest temporally to the sectored data.

  • False if the current area definition is removed when filtering the list of area definitions based on the actual sectored data time.

Return type:

bool

Module contents#

GeoIPS procflow init file.

On this page
Show Source