geoips.utils.memusg package#

Submodules#

geoips.utils.memusg.boxplots module#

Functions for creating box and whisker plots.

geoips.utils.memusg.boxplots.boxplot(memusg_dict, stat='Max. Resident Set Size (GB)', rotation=90, fontsize=12, figsize=(8, 8), skip_sources=['modis'], geoips_vers=None, output_dir=None, window_start_datetime=None, window_end_datetime=None, source_map=None)[source]#

Create box and whisker plots of overall resource usage for procflows.

geoips.utils.memusg.boxplots.checkpoint_boxplot(memusg_dict, stat='elapsed time', rotation=90, fontsize=18, figsize=(24, 8), source=None, platform=None, skip_sources=['modis'], skip_sectors=['DYNAMIC-TC', 'DYNAMIC TC'], plot_sectors=None, geoips_vers=None, output_dir=None, window_start_datetime=None, window_end_datetime=None, source_map=None)[source]#

Create box and whisker plots of stats broken down by checkpoints.

geoips.utils.memusg.dataframe_utils module#

Various utils for processing statistics.

geoips.utils.memusg.dataframe_utils.create_checkpoint_dataframe(json_data, procflow_usg)[source]#

Create a checkpoint dataframe that incorporates stats from overall usage stats.

Parameters:

  • json_data (list) – List of json data entries

  • procflow_usg (dict) – Dict of panda dataframes where each key is a different source

Returns:

dict of panda dataframes where each key is a different source

Return type:

dict

geoips.utils.memusg.dataframe_utils.print_stats(dframe, suite_name=' ')[source]#

Print stats to stdout.

Parameters:

  • dframe (pandas.core.frame.DataFrame) – DataFrame of statistics

  • suite_name (str, optional) – Source name of stats, by default “ “

geoips.utils.memusg.dataframe_utils.sort_resource_dframe(dframe)[source]#

Sort resource usage statistics by sector type.

Parameters:

dframe (pandas.core.frame.DataFrame) – Stats

Returns:

Dictionary of dataframes where each key is a different sector type

Return type:

dict

geoips.utils.memusg.dataframe_utils.sort_resource_stats(resource_stats_dict, sort_platform=False)[source]#

Sort resource usage for each source by different sector types.

Parameters:

  • resource_stats_dict (dict) – Dictionary of dataframes for each source

  • sort_platform (bool, optional) – Separate dataframes by platform in addition to source, by default False

Returns:

dictionary of dataframes

Return type:

dict

geoips.utils.memusg.memusg_tracker module#

Utilities for tracking and monitoring memory and resource usage.

class geoips.utils.memusg.memusg_tracker.PidLog(inpid, logstr='')[source]#

Bases: object

Track a PID and all children.

  • Requires psutil and threading

checkpoint_usage_stats(write_to_json=False, metadata=None)[source]#

Return organized dictionary of stats from track_resource_usage.

Returns:

Resource usage statistics for markers/checkpoints recorded by the track_resource_usage method. Dictionary is ordered by the markers passed to track_resource_usage. Each key in the return dictionary will hold a dictionary of the resource usage statistics. If a group is specified in the marker name (e.g. “FOO: BAR”), the return dictionary will have a “BAR” key, and will have a “checkpoint_group” key in the corresponding resource usage dictionary with a value of “FOO”.

Return type:

dict

print_mem_usg(logstr='', verbose=False, write_to_json=False, metadata=None)[source]#

Print verbose resouce usage.

print_resource_usage()[source]#

Print verbose resource usage, using “resource” package.

save_csv()[source]#

Save a csv file to output.

save_exit()[source]#

Exit the thread cleanly.

track_pids()[source]#

Track pids and create a dict of values.

track_resource_usage(logstr='', verbose=False, key=None, show_log=True, checkpoint=False, increment_key=True)[source]#

Record resouce usage for a given processing marker in GeoIPS.

Parameters:

  • logstr (str, optional) – String to include at the start of the log message, by default “”

  • verbose (bool, optional) – Print the full resource usage statistics dict to stdout, by default False

  • key (str, optional) – Unique name of marker to record statistics, by default None A marker can be categorized as part of a group when using “:” in the marker name (e.g “FOO: BAR”). This is used in the checkpoint_usage_stats method, where markers will have a “checkpoint_group” key in their resource usage statistics dictionary holding the name of the group. (e.g. stats[“BAR”][“checkpoint_group”] == “FOO”)

  • show_log (bool, optional) – Use LOG.info to print the current memory usage, by default True

  • checkpoint (bool, optional) – Re-use a marker key to record more detailed profiling of resource usage in terms of both timing and maximum memory usage, by default False

  • increment_key (bool, optional) – Append the number of times marker is re-used to the key, by default True Marker names should be unique, and should at minimum hold the start/end resource usage for the marker. If a marker is re-used and increment_key is False, a KeyError is raised. If increment_key is True and the marker has both start/end stats, a number is automatically appended to the key name. This number represents how many times a marker has been re-used. For example: “FOO: BAR” -> “FOO: BAR(1)”: -> “FOO: BAR(2)” (Note: “FOO: BAR(2)” is only created if “FOO: BAR(1)” has start/end stats)

Returns:

Resource usage statistics for checkpoint

Return type:

dict

Raises:

KeyError – Duplicate key is discovered and increment_key is False

geoips.utils.memusg.memusg_tracker.print_mem_usage(logstr='', verbose=False)[source]#

Print memory usage to LOG.info.

  • By default include psutil output.

  • If verbose is True, include output from both psutil and resource packages.

geoips.utils.memusg.memusg_tracker.print_resource_usage(logstr='')[source]#

Print verbose resource usage, using “resource” package.

geoips.utils.memusg.memusg_tracker.single_track_pid(procpid)[source]#

Output a snapshot of a pid usage on server.

  • Requires an input pid.

geoips.utils.memusg.plot_json_files module#

Find JSON statistics files on disk and plot stats.

geoips.utils.memusg.plot_json_files.create_usage_dataframe_per_source(loaded_json_dataframe)[source]#

Separate dataframe into multiple by source.

Parameters:

loaded_json_dataframe (pandas.core.frame.DataFrame) – Single dataframe with all stats

Returns:

Dict of DataFrames where each key is a different source

Return type:

dict

geoips.utils.memusg.plot_json_files.get_json_file_lists(json_stats_output_directory, geoips_version, window_start, window_end, sector_type=None, include_checkpoints=False)[source]#

Find list of JSON statistic files to load.

Parameters:

  • json_stats_output_directory (str) – Base output directory of JSON files

  • geoips_version (str) – Version of GeoIPS

  • window_start (datetime.datetime) – Start of search window

  • window_end (datetime.datetime) – End of search window

  • sector_type (str, optional) – Limit jsons to either: mixed, static, or tc specific procflows, by default None

  • include_checkpoints (bool, optional) – Attempt to load detailed checkpoint statistics, by default False

Returns:

List of files for each key

Return type:

dict

geoips.utils.memusg.plot_json_files.load_jsons_into_dataframes_by_source(json_files, json_filename_keys=None)[source]#

Take list of json files and return dict of DataFrames.

Parameters:

  • json_files (list) – List of json files to read

  • json_filename_keys (list, optional) – List of filename tags used to differentiate experiments

Returns:

Dict of DataFrames where each key is a different source

Return type:

dict

geoips.utils.memusg.plot_json_files.load_jsons_into_single_dataframe(json_files, json_filename_keys=None, rename_columns=False)[source]#

Load JSON files into single pandas dataframe.

Parameters:

  • json_files (list) – List of json files to read

  • json_filename_keys (list, optional) – List of filename tags used to differentiate experiments

  • rename_columns (bool, optional) – Rename columns to be more plotting friendly, by default False

Returns:

Dataframe of stats

Return type:

pandas.core.frame.DataFrame

geoips.utils.memusg.plot_json_files.main()[source]#

Plot statistics.

Module contents#

Utilities for memory usage tracking and reporting.

class geoips.utils.memusg.PidLog(inpid, logstr='')[source]#

Bases: object

Track a PID and all children.

  • Requires psutil and threading

checkpoint_usage_stats(write_to_json=False, metadata=None)[source]#

Return organized dictionary of stats from track_resource_usage.

Returns:

Resource usage statistics for markers/checkpoints recorded by the track_resource_usage method. Dictionary is ordered by the markers passed to track_resource_usage. Each key in the return dictionary will hold a dictionary of the resource usage statistics. If a group is specified in the marker name (e.g. “FOO: BAR”), the return dictionary will have a “BAR” key, and will have a “checkpoint_group” key in the corresponding resource usage dictionary with a value of “FOO”.

Return type:

dict

print_mem_usg(logstr='', verbose=False, write_to_json=False, metadata=None)[source]#

Print verbose resouce usage.

print_resource_usage()[source]#

Print verbose resource usage, using “resource” package.

save_csv()[source]#

Save a csv file to output.

save_exit()[source]#

Exit the thread cleanly.

track_pids()[source]#

Track pids and create a dict of values.

track_resource_usage(logstr='', verbose=False, key=None, show_log=True, checkpoint=False, increment_key=True)[source]#

Record resouce usage for a given processing marker in GeoIPS.

Parameters:

  • logstr (str, optional) – String to include at the start of the log message, by default “”

  • verbose (bool, optional) – Print the full resource usage statistics dict to stdout, by default False

  • key (str, optional) – Unique name of marker to record statistics, by default None A marker can be categorized as part of a group when using “:” in the marker name (e.g “FOO: BAR”). This is used in the checkpoint_usage_stats method, where markers will have a “checkpoint_group” key in their resource usage statistics dictionary holding the name of the group. (e.g. stats[“BAR”][“checkpoint_group”] == “FOO”)

  • show_log (bool, optional) – Use LOG.info to print the current memory usage, by default True

  • checkpoint (bool, optional) – Re-use a marker key to record more detailed profiling of resource usage in terms of both timing and maximum memory usage, by default False

  • increment_key (bool, optional) – Append the number of times marker is re-used to the key, by default True Marker names should be unique, and should at minimum hold the start/end resource usage for the marker. If a marker is re-used and increment_key is False, a KeyError is raised. If increment_key is True and the marker has both start/end stats, a number is automatically appended to the key name. This number represents how many times a marker has been re-used. For example: “FOO: BAR” -> “FOO: BAR(1)”: -> “FOO: BAR(2)” (Note: “FOO: BAR(2)” is only created if “FOO: BAR(1)” has start/end stats)

Returns:

Resource usage statistics for checkpoint

Return type:

dict

Raises:

KeyError – Duplicate key is discovered and increment_key is False

geoips.utils.memusg.print_mem_usage(logstr='', verbose=False)[source]#

Print memory usage to LOG.info.

  • By default include psutil output.

  • If verbose is True, include output from both psutil and resource packages.