geoips.plugins.modules.procflows package#

Submodules#

geoips.plugins.modules.procflows.config_based module#

Processing workflow for config-based processing.

geoips.plugins.modules.procflows.config_based.call(fnames, command_line_args=None)[source]#

Workflow for efficiently running all required outputs.

Includes all sectors and products specified in a YAML output config file. Specified via a YAML config file

Parameters:

  • fnames (list) – List of strings specifying full paths to input file names to process

  • command_line_args (dict) – dictionary of command line arguments

Returns:

0 for successful completion, non-zero for error (incorrect comparison, or failed run)

Return type:

int

geoips.plugins.modules.procflows.config_based.get_area_def_list_from_dict(area_defs)[source]#

Get a list of actual area_defs from full dictionary.

Dict returned from get_area_defs_from_available_sectors

geoips.plugins.modules.procflows.config_based.get_area_defs_from_available_sectors(available_sectors_dict, command_line_args, xobjs, variables)[source]#

Get all required area_defs for the given set of parameters.

YAML config parameters (config_dict), command_line_args, xobjs, and required variables. Command line args override config specifications.

Parameters:

  • available_sectors_dict (dict) – Dictionary of all requested sector_types (specified in YAML config)

  • command_line_args (dict) – Dictionary of command line arguments - any command line argument that is also a key in available_sectors_dict[<sector_type>] will replace the value in the available_sectors_dict[<sector_type>]

  • xobjs (dict) – Dictionary of xarray datasets, used in determining start/end time of data files for identifying dynamic sectors

  • variables (list) – List of required variables, for determining center coverage for TCs

Returns:

Dictionary of required area_defs, with area_def.name as the dictionary keys. Based on YAML config-specified available_sectors, and command line args

Return type:

dict

Notes

  • Each area_def.name key has one or more “sector_types” associated with it.

  • Each sector_type dictionary contains the actual “requested_sector_dict” from the YAML config, and the actual AreaDefinition object that was returned.

    • area_defs[area_def.name][sector_type]['requested_sector_dict']

    • area_defs[area_def.name][sector_type]['area_def']

geoips.plugins.modules.procflows.config_based.get_bg_xarray(sect_xarrays, area_def, prod_plugin, resampled_read=False)[source]#

Get background xarray.

geoips.plugins.modules.procflows.config_based.get_config_dict(config_yaml_file)[source]#

Populate the full config dictionary from a given YAML config file.

Includes both sector and output specifications.

Parameters:

config_yaml_file (str) – Full path to YAML config file, containing sector and output specifications. YAML config files support environment variables in entries flagged with !ENV

Returns:

Return dictionary of both sector and output specifications, as found in config_yaml_file. The output dictionary references the “sector_types” found in the available_sectors dictionary, each output_type requests a specific “sector_type” to be used for processing.

Return type:

dict

geoips.plugins.modules.procflows.config_based.get_required_outputs(config_dict, sector_type)[source]#

Get only the required outputs from the current sector_type.

geoips.plugins.modules.procflows.config_based.get_resampled_read(config_dict, area_defs, area_def_id, sector_type, reader, fnames, variables)[source]#

Return dictionary of xarray datasets for a given area def.

Xarrays resampled to area_def

geoips.plugins.modules.procflows.config_based.get_sectored_read(config_dict, area_defs, area_def_id, sector_type, reader, fnames, variables)[source]#

Return dictionary of xarray datasets for a given area def.

Xarrays sectored to area_def

geoips.plugins.modules.procflows.config_based.get_variables_from_available_outputs_dict(available_outputs_dict, source_name, sector_types=None)[source]#

Get required variables for all outputs for a given “source_name”.

Outputs specified within the YAML config.

Parameters:

  • available_outputs_dict (dict) – Dictionary of all requested output_types (specified in YAML config)

  • source_name (str) – Find all required variables for the passed “source_name”

  • sector_types (list, default=None) – if sector_types list of strings is passed, only include output_types that require one of the passed “sector_types”

Returns:

List of all required variables for all output products for the given source_name

Return type:

list

geoips.plugins.modules.procflows.config_based.initialize_final_products(final_products, cpath, cmodule)[source]#

Initialize the final_products dictionary with cpath dict key if needed.

Parameters:

  • final_products (dict) – Dictionary of final products, with keys of final required “compare_path” Products with no compare_path specified are stored with the key “no_comparison”

  • cpath (str) – Key to add to final_products dictionary

Returns:

Return final_products dictionary, updated with current “cpath” key: final_products[cpath][‘files’] = <list_of_files_in_given_cpath>

Return type:

dict

geoips.plugins.modules.procflows.config_based.is_required_sector_type(available_outputs_dict, sector_type)[source]#

Check if current sector is required for any outputs.

Check if a given sector_type is required for any currently requested output_types

Parameters:

  • available_outputs_dict (dict) – Dictionary of all requested output_types (specified in YAML config)

  • sector_type (str) – Determine if any output_types require the currently requested “sector_type”

Returns:

  • True if any output_types require the passed “sector_type”

  • False if no output_types require the passed “sector_type”

Return type:

bool

geoips.plugins.modules.procflows.config_based.process_unsectored_data_outputs(final_products, available_outputs_dict, available_sectors_dict, xobjs, variables, command_line_args=None, write_to_product_db=False)[source]#

Process unsectored data output.

Loop through all possible outputs, identifying output types that require unsectored data output. Produce all required unsectored data output, update final_products dictionary accordingly, and return final_products dictionary with the new unsectored outputs.

Parameters:

  • final_products (dict) – Dictionary of final products, with keys of final required “compare_path” Products with no compare_path specified are stored with the key “no_comparison”

  • available_outputs_dict (dict) – Dictionary of all available output product specifications

  • available_sectors_dict (dict) – Dictionary of available sector types - we are looking for available sectors that contain the “unsectored” keyword.

  • xobjs (dict) – Dictionary of xarray datasets, for use in producing unsectored output formats

  • variables (list) – List of strings of required variables in the given product.

Returns:

Return final_products dictionary, updated with current “cpath” key: final_products[cpath][‘files’] = <list_of_files_in_given_cpath>

Return type:

dict

geoips.plugins.modules.procflows.config_based.requires_bg(available_outputs_dict, sector_type)[source]#

Check if current sector requires background imagery.

Check if a given sector_type is requested for any product_types that also require background imagery.

Parameters:

  • available_outputs_dict (dict) – Dictionary of all requested output_types (specified in YAML config)

  • sector_type (str) – sector_type to determine if any output_types that require background imagery also request the passed sector_type

Returns:

  • True if any output_types that require background imagery require the passed “sector_type”

  • False if no output_types require both background imagery and the passed “sector_type”

Return type:

bool

geoips.plugins.modules.procflows.config_based.set_comparison_path(output_dict, product_name, output_type, command_line_args=None)[source]#

Replace variables specified by <varname> in compare_path.

Parameters:

  • config (dict) – Dictionary of output specifications, containing key “compare_path”

  • product_name (str) – Current requested product name, all instances of <product> in compare_path replaced with product_name argument

  • output_type (str) – Current requested output type, all instances of <output> in compare_path replaced with output argument

Returns:

Return a single string with the fully specified comparison path for current product

Return type:

str

geoips.plugins.modules.procflows.config_based.update_output_dict_from_command_line_args(output_dict, command_line_args=None)[source]#

Update output dict from command line args.

geoips.plugins.modules.procflows.single_source module#

Processing workflow for single data source processing.

geoips.plugins.modules.procflows.single_source.add_filename_extra_field(xarray_obj, field_name, field_value)[source]#

Add filename extra field.

geoips.plugins.modules.procflows.single_source.call(fnames, command_line_args=None)[source]#

Workflow for running products from a single data source.

Parameters:

  • fnames (list) – List of strings specifying full paths to input file names to process

  • command_line_args (dict) – dictionary of command line arguments

Returns:

Return list of strings specifying full paths to output products that were produced

Return type:

list

See also

geoips.commandline.args

Complete list of available command line args.

geoips.plugins.modules.procflows.single_source.combine_filename_extra_fields(source_xarray, dest_xarray)[source]#

Combine filename extra fields.

geoips.plugins.modules.procflows.single_source.get_alg_xarray(sect_xarrays, area_def, prod_plugin, resector=True, resampled_read=False, variable_names=None)[source]#

Get alg xarray.

geoips.plugins.modules.procflows.single_source.get_area_defs_from_command_line_args(command_line_args, xobjs, variables=None, filter_time=True)[source]#

Get area def from command line args.

geoips.plugins.modules.procflows.single_source.get_filename(filename_formatter, prod_plugin=None, alg_xarray=None, area_def=None, supported_filenamer_types=None, output_dict=None, filename_formatter_kwargs=None)[source]#

Get filename.

geoips.plugins.modules.procflows.single_source.get_output_filenames(fname_formats, output_dict, prod_plugin, xarray_obj=None, area_def=None, supported_filenamer_types=None)[source]#

Get output filenames.

geoips.plugins.modules.procflows.single_source.output_all_metadata(output_dict, output_fnames, metadata_fnames, xarray_obj, area_def=None)[source]#

Output all metadata.

geoips.plugins.modules.procflows.single_source.pad_area_definition(area_def, source_name=None, force_pad=False, x_scale_factor=1.5, y_scale_factor=1.5)[source]#

Pad area definition.

geoips.plugins.modules.procflows.single_source.plot_data(output_dict, alg_xarray, area_def, prod_plugin, output_kwargs, fused_xarray_dict=None, no_output=False)[source]#

Plot data.

alg_xarray used for filename formats, etc. If included, fused_xarray_dict used for output format call

geoips.plugins.modules.procflows.single_source.print_area_def(area_def, print_str)[source]#

Print area def.

geoips.plugins.modules.procflows.single_source.process_sectored_data_output(xobjs, variables, prod_plugin, output_dict, area_def=None)[source]#

Process sectored data output.

If product family is ‘sectored_xarray_dict_to_output_format’, call ‘process_xarray_dict_to_output_format’, store the result in a list, and return it.

geoips.plugins.modules.procflows.single_source.process_xarray_dict_to_output_format(xobjs, variables, prod_plugin, output_dict, area_def=None)[source]#

Process xarray dict to output format.

geoips.plugins.modules.procflows.single_source.remove_unsupported_kwargs(module, requested_kwargs)[source]#

Remove unsupported keyword arguments.

geoips.plugins.modules.procflows.single_source.verify_area_def(area_defs, check_area_def, data_start_datetime, data_end_datetime, time_range_hours=3)[source]#

Verify current area definition is the closest to the actual data time.

When looping through multiple dynamic area definitions for a full data file that temporally covers more than one dynamic area_def, there is no way of knowing which dynamic area_def has the best coverage until AFTER we have actually sectored the data to the specific area_def.

Call this utility on the current area_def (check_area_def) for the sectored data file, plus the full list of area definitions (area_defs) that cover the FULL data file.

Returns:

  • True if the current area definition is NOT dynamic

  • True if the current area definition IS dynamic and is the closest temporally to the sectored data.

  • False if the current area definition is removed when filtering the list of area definitions based on the actual sectored data time.

Return type:

bool

Module contents#

Geoips procflow init file.