geoips.utils package

Subpackages

• geoips.utils.coverage_checkers package
  • Module contents
    • create_radius()
• geoips.utils.memusg package
  • Submodules
  • geoips.utils.memusg.boxplots module
    • boxplot()
    • checkpoint_boxplot()
  • geoips.utils.memusg.dataframe_utils module
    • create_checkpoint_dataframe()
    • print_stats()
    • sort_resource_dframe()
    • sort_resource_stats()
  • geoips.utils.memusg.memusg_tracker module
    • PidLog
      • PidLog.checkpoint_usage_stats()
      • PidLog.print_mem_usg()
      • PidLog.print_resource_usage()
      • PidLog.save_csv()
      • PidLog.save_exit()
      • PidLog.track_pids()
      • PidLog.track_resource_usage()
    • print_mem_usage()
    • print_resource_usage()
    • single_track_pid()
  • geoips.utils.memusg.plot_json_files module
    • create_usage_dataframe_per_source()
    • get_json_file_lists()
    • load_jsons_into_dataframes_by_source()
    • load_jsons_into_single_dataframe()
    • main()
  • Module contents
    • PidLog
      • PidLog.checkpoint_usage_stats()
      • PidLog.print_mem_usg()
      • PidLog.print_resource_usage()
      • PidLog.save_csv()
      • PidLog.save_exit()
      • PidLog.track_pids()
      • PidLog.track_resource_usage()
    • print_mem_usage()
• geoips.utils.types package
  • Submodules
  • geoips.utils.types.converter_registry module
    • TypeConverterRegistry
      • TypeConverterRegistry.can_convert()
      • TypeConverterRegistry.convert()
      • TypeConverterRegistry.register()
      • TypeConverterRegistry.register_bidirectional()
      • TypeConverterRegistry.registered_types
  • geoips.utils.types.converters module
    • Metadata policy
    • FamilyConversionSpec
      • FamilyConversionSpec.input_converter
      • FamilyConversionSpec.input_type
      • FamilyConversionSpec.output_converter
      • FamilyConversionSpec.output_type
    • dataarray_to_dataset()
    • dataset_dict_to_dataset()
    • dataset_to_dataarray()
    • dataset_to_dataset_dict()
    • dataset_to_dict()
    • dataset_to_list()
    • dataset_to_masked_array()
    • dataset_to_numpy()
    • dataset_vars_to_numpy_list()
    • dict_to_dataset()
    • list_to_dataset()
    • masked_array_to_dataset()
    • numpy_to_dataset()
  • geoips.utils.types.datatree_converters module
  • geoips.utils.types.datatree_ditto module
    • DataTreeDitto
      • DataTreeDitto.filter()
      • DataTreeDitto.from_datatree()
      • DataTreeDitto.get()
      • DataTreeDitto.get_original()
      • DataTreeDitto.items()
      • DataTreeDitto.keys()
      • DataTreeDitto.list_converted_nodes()
      • DataTreeDitto.map_over_datasets()
      • DataTreeDitto.match()
      • DataTreeDitto.mean()
      • DataTreeDitto.to_datatree()
      • DataTreeDitto.values()
  • geoips.utils.types.datatree_helpers module
    • has_data_vars()
    • to_mutable_dataset()
  • geoips.utils.types.family_conversions module
  • geoips.utils.types.obp_conduits module
    • OBP_CONDUITS
    • kwarg_name_for_kind()
  • geoips.utils.types.partial_lexeme module
    • Lexeme
      • Lexeme.plural
      • Lexeme.singular
  • geoips.utils.types.script_datatree module
    • RetentionPolicy
      • RetentionPolicy.current_only
      • RetentionPolicy.keep_all
      • RetentionPolicy.metadata_only
    • add_data_step()
    • apply_script_retention()
    • attach_plugin_result()
    • attach_script_result()
    • get_current_data()
    • get_output_products()
    • initialize_script_tree()
    • is_script_tree()
    • normalize_retention_policy()
    • plugin_kind()
    • script_call_data()
    • validate_retention_policy()
  • geoips.utils.types.tokenization module
    • compute_arguments_hash()
    • compute_step_output_token()
    • compute_token()
  • geoips.utils.types.yaml_plugin_callable module
    • YamlPluginCallable
      • YamlPluginCallable.call()
      • YamlPluginCallable.spec
  • Module contents

Submodules

geoips.utils.cache_files module

Module for handling cached files in GeoIPS.

This modules provides functions to manage cache files in GeoIPS. Cache files will be stored in the user cache directory, which is platform-dependent. The correct cache directory is determined using the GEOIPS_CACHE_DIR environment variable which defaults to platformdirs.user_cache_dir(“geoips”) if not set.

geoips.utils.cache_files.create_cached_json_from_yaml(source, cache_dir=None)

Create a cached JSON file from a YAML file.

This function reads a YAML file and writes its contents to a JSON file in the user cache directory. The JSON file will be created if it does not already exist, or updated if the source YAML file’s contents have changed since it was last cached.

Parameters:

• source (str) – The path to the source YAML file.

• cache_dir (str, optional) – The path to the cache directory. If not provided, the default user cache directory will be used.

Returns:

The path to the cached JSON file.

Return type:

str

geoips.utils.cache_files.get_cached_json(source, cache_dir=None)

Get the cached JSON file corresponding to a YAML file.

Some files in GeoIPS are stored in YAML format, but we want to use JSON at runtime because loading JSON is faster than loading YAML. This function checks if the cached JSON file exists and is up to date. If it does not exist or is out of date, it creates a new cached JSON file from the YAML file. It then returns the contents of the cached JSON file.

Parameters:

• source (str) – The path to the source YAML file.

• cache_dir (str, optional) – The path to the cache directory. If not provided, the default user cache directory will be used.

geoips.utils.cache_files.source_modified(source, dest)

Check if the source file was modified more recently than the destination file.

This uses os.path.getmtime() to determine whether the source file has been modified more recently than the destination file. Returns True if the source file was modified more recently than the destination file, False otherwise.

Parameters:

• source (str) – The path to the source file we are monitoring for changes.

• dest (str) – The path to the destination file we will update if the source file changes.

Returns:

True if the source file was modified more recently than the destination file, False otherwise.

Return type:

bool

geoips.utils.composite module

Utilities for swath compositing in GeoIPS.

geoips.utils.composite.find_preproc_alg_files(product_time, composite_window, sector_name, product, sensor, platform, file_format='netcdf', product_db=False, db_query_plugin=None, db_schemas=None, db_tables=None)

Find pre-processed algorithm files that were saved to disk.

Parameters:

• product_time (datetime.datetime) – Product time

• composite_window (str) – How far back to search for pre-processed files. Window needs to be specified in iso8601 duration format (e.g. PT4H)

• sector_name (str) – Name of sector to composite

• product (str) – Name of product to composite

• sensor (str) – Name of sensor to composite

• platform (str) – Name of platform to composite

• file_format (str, optional) – Pre-processed file format, by default “netcdf”

• product_db (bool, optional) – Use product database to find any pre-processed file, by default False

• db_query_plugin (str, optional) – Name of product database query plugin, by default None

• db_schemas (list, optional) – Names of postgres schema to query, by default None

• db_tables (list, optional) – Names of table to query under schema, by default None

Returns:

List of pre-processed algorithm files

Return type:

list

geoips.utils.composite.find_preproc_alg_netcdfs(product_time_start, product_time_end, sector_name, product, sensor, platform, product_db=False, postgres_query_plugin=None, postgres_schemas=None, postgres_tables=None)

Find pre-processed algorithm netCDF files that were saved to disk.

Parameters:

• product_time_start (datetime.datetime) – Earliest product time to search for valid files

• product_time_start – Latest product time to search for valid files

• sector_name (str) – Name of sector to composite

• product (str) – Name of product to composite

• sensor (str) – Name of sensor to composite

• platform (str) – Name of platform to composite

• file_format (str, optional) – Pre-processed file format, by default “netcdf”

• product_db (bool, optional) – Use product database to find any pre-processed file, by default False

• db_query_plugin (str, optional) – Name of product database query plugin, by default None

• db_schemas (list, optional) – Names of postgres schema to query, by default None

• db_tables (list, optional) – Names of table to query under schema, by default None

Returns:

List of pre-processed netCDF algorithm files

Return type:

list

geoips.utils.context_managers module

Module for handling optional dependencies throughout GeoIPS.

geoips.utils.context_managers.import_optional_dependencies(loglevel='info')

Attempt to import a package and log the event if the import fails.

Parameters:

loglevel (str) – Name of the log level to write to. May be any valid log level (e.g. debug, info, etc.).

geoips.utils.decorators module

GeoIPS decorators module.

class geoips.utils.decorators.deprecated(replacement=None)

Bases: object

A decorator that deprecates a function.

When applied to a function, will cause that function to raise a DeprecationWarning when called.

geoips.utils.decorators.developmental(func)

Mark an interfaces as developmental.

When applied to a function, will prepend a “developmental” message to the beginning of that function’s docstring.

geoips.utils.file_utils module

Filesystem path helpers used across GeoIPS.

geoips.utils.file_utils.path_exists(path: str | Path) → bool

Return True if a filesystem path exists.

Parameters:

file_path (str) –

File or Directory path to evaluate. This could be a file, directory, or other

filesystem object.

Returns:

True if path exists on the filesystem, otherwise False.

Return type:

bool

Module contents

Geoips utilities init file.