pluginify package#

Subpackages#

Submodules#

pluginify.commandline_typer module#

Commandline module for Pluginify.

Supports two main commands, pluginify create and pluginify delete as well as configuration commands pluginify config set-namespace, pluginify config set-rebuild-registries and pluginify config set-registry-directory.

The main commands create / delete plugin registries for one or more packages under a given namespace of a certain file type [.json, .yaml]. If deleting, both file types are deleted if found.

The config commands set configuration variables which direct pluginify where to look for plugins, whether or not it should rebuild registries by default, and where it should write registry files to.

class pluginify.commandline_typer.DocstringTyper(*, name: str | None = <typer.models.DefaultPlaceholder object>, cls: type[typer.core.TyperGroup] | None = <typer.models.DefaultPlaceholder object>, invoke_without_command: bool = <typer.models.DefaultPlaceholder object>, no_args_is_help: bool = <typer.models.DefaultPlaceholder object>, subcommand_metavar: str | None = <typer.models.DefaultPlaceholder object>, chain: bool = <typer.models.DefaultPlaceholder object>, result_callback: ~collections.abc.Callable[[...], ~typing.Any] | None = <typer.models.DefaultPlaceholder object>, context_settings: dict[typing.Any, typing.Any] | None = <typer.models.DefaultPlaceholder object>, callback: ~collections.abc.Callable[[...], ~typing.Any] | None = <typer.models.DefaultPlaceholder object>, help: str | None = <typer.models.DefaultPlaceholder object>, epilog: str | None = <typer.models.DefaultPlaceholder object>, short_help: str | None = <typer.models.DefaultPlaceholder object>, options_metavar: str = <typer.models.DefaultPlaceholder object>, add_help_option: bool = <typer.models.DefaultPlaceholder object>, hidden: bool = <typer.models.DefaultPlaceholder object>, deprecated: bool = <typer.models.DefaultPlaceholder object>, add_completion: bool = True, rich_markup_mode: ~typing.Literal['markdown', 'rich', None] = 'rich', rich_help_panel: str | None = <typer.models.DefaultPlaceholder object>, suggest_commands: bool = True, pretty_exceptions_enable: bool = True, pretty_exceptions_show_locals: bool = False, pretty_exceptions_short: bool = True)[source]#

Bases: Typer

Subclass of typer.Typer that generates help messages from a command’s docstring.

Provided that a docstring has a ‘parameters’ section, all help messages for required or optional arguments will be automatically generated.

Additionally, shorthand flags will be generated for all flag-based arguments. I.e. ‘–namespace’ automatically generates a ‘-n’ flag.

command(*args, **kwargs)[source]#: Overridden typer.Typer:command decorator.

pluginify.commandline_typer.configure_logging(level=20)[source]#

Configure root logging for the command-line interface.

This function attaches a StreamHandler to the root logger so that all log messages emitted by this process—including logs from imported modules, dynamically loaded plugins, and third-party libraries—are routed to the terminal.

Logging is configured only once. If handlers are already attached to the root logger, this function exits without making changes to avoid duplicate log output.

Parameters:: level (int, optional) – Logging level to apply to the root logger and handler. Defaults to logging.INFO. Common values include logging.DEBUG, logging.INFO, logging.WARNING, and logging.ERROR.
Return type:: None

pluginify.commandline_typer.create(namespace: str = <typer.models.OptionInfo object>, packages: ~typing.List[str] | None = <typer.models.OptionInfo object>, save_type: ~typing.Literal['json', 'yaml'] = <typer.models.OptionInfo object>)[source]#

Create plugin registries.

Plugin registries are low-level files which contain dictionaries of metadata for all plugins implemented in one or more packages registered under a common namespace.

Pluginify creates and makes use of these registry files to properly load and instantiate your plugins.

Examples

Namespace: packageX.plugin_packages Directory structure:

packageX/
plugins/ interfaces/ utils/ registered_plugins.json [will create]

packageY/
plugins/ registered_plugins.json [will create]

JSON registry files are faster to load and are used by default. YAML extensions are also supported for easier viewing.

CLI examples#

pluginify create pluginify create -n packageX.plugin_packages pluginify create -s yaml pluginify create -n packageX.plugin_packages -p packageY

pluginify.commandline_typer.delete(namespace: str = <typer.models.OptionInfo object>, packages: ~typing.List[str] | None = <typer.models.OptionInfo object>)[source]#

Delete plugin registries.

Plugin registries are low-level files which contain dictionaries of metadata for all plugins implemented in one or more packages registered under a common namespace.

By default, this command will delete every .json and .yaml instance of registry files for every package registered under a given namespace. If you only want to delete registry files from a subset of packages, use the ‘-p’ flag to specify those packages.

Examples

Namespace: packageX.plugin_packages Directory structure:

packageX/
plugins/ interfaces/ utils/ registered_plugins.json [will delete]

packageY/
plugins/ registered_plugins.json [will delete]

CLI examples#

pluginify delete pluginify delete -n packageX.plugin_packages pluginify delete -n packageX.plugin_packages -p packageY

pluginify.commandline_typer.main()[source]#: Entrypoint function for pluginify’s CLI.

pluginify.commandline_typer.set_namespace(namespace: str = <typer.models.ArgumentInfo object>)[source]#: Set pluginify’s NAMESPACE config variable.

pluginify.commandline_typer.set_rebuild_registries(rebuild_registries: bool = <typer.models.ArgumentInfo object>)[source]#: Set pluginify’s REBUILD_REGISTRIES config variable.

pluginify.commandline_typer.set_registry_directory(registry_directory: ~pathlib.Path = <typer.models.ArgumentInfo object>)[source]#: Set pluginify’s REGISTRY_DIRECTORY config variable.

pluginify.commandline_typer.update_existing_fields(new_data)[source]#

Overwrite fields in pluginify’s config file if they exist in ‘new_data’.

Parameters:: new_data (dict) – A dictionary containing configuration key, value pairs to overwrite data in config_path.

pluginify.config module#

Module containing configuration variables needed for pluginify to run.

pluginify.config.get_registry_cache_dir(namespace, package)[source]#

Return the path to the parent directory of where to write registry files to.

Where the path is formatted ‘~/.cache/{env_name}/{namespace}/{package}’.

Parameters:

namespace (str) – Namespace that your plugin packages fall under. The argument parser defaults this value to ‘pluginify.plugin_packages’, but a user can create separate namespaces if developing interfaces outside of pluginify.
package (str) – Name of a plugin package that is registered under ‘namespace’.

Returns:

cache_dir – Full path to the parent directory in which registry files should be written for a package under namespace.

Return type:

Path

pluginify.create_plugin_registries module#

Generates all available plugins from installed packages under a given namespace.

After all plugins have been generated, they are written to a registered_plugins.json file which contains a dictionary of all the registered plugins across all plugin repositories under a given namespace.

Run ‘pluginify create’ to produce registered_plugins.json for EVERY currently installed plugin package under a given namespace (default=’pluginify.plugin_packages’). A separate registered_plugins.json is created at the top level package directory for each plugin package under that namespace.

pluginify.create_plugin_registries.add_class_plugin(package, relpath, plugins)[source]#

Add the class-based plugin associated with the filepaths and package to plugins.

NOTE: This function will work for ‘legacy’ module-based plugins that are dynamically converted to class-based objects via <interface>.get_plugin and true class-based plugins. Stores them all under the ‘class_based’ portion of the plugin registry and denotes whether a plugin is truly class based or not via the metadata variable ‘is_derived_from_module’.

Parameters:

package (str) – The current package being parsed
relpath (str) – The relpath path to the class-based plugin
plugins (dict) – A dictionary object of all installed package plugins

Returns:

error_message – String containing informative error messages from any plugins that were improperly formatted. An exception will be raised at the very end if error_message is not the empty string - this allows collecting ALL errors throughout the plugin registry process and reporting them all at once, to facilitate rapidly identifying and resolving errors.

Return type:

str

pluginify.create_plugin_registries.add_text_plugin(package, relpath, plugins)[source]#

Add all text plugins into plugin registries.

Parameters:

package (str) – The current package being parsed
relpath (str) – The relpath path to the text plugin
plugins (dict) – A dictionary object of all installed package plugins

Returns:

error_message – String containing informative error messages from any plugins that were improperly formatted. An exception will be raised at the very end if error_message is not the empty string - this allows collecting ALL errors throughout the plugin registry process and reporting them all at once, to facilitate rapidly identifying and resolving errors.

Return type:

str

pluginify.create_plugin_registries.add_yaml_plugin(filepath, relpath, package, plugins, namespace)[source]#

Add the yaml plugin associated with the filepaths and package to plugins.

Parameters:

filepath (str) – The path of the plugin derived from resouces.files(package) / plugin
relpath (str) – The relative path to the filepath provided
package (str) – The current package being parsed
plugins (dict) – A dictionary object of all installed package plugins
namespace (str) – Namespace that your plugin packages fall under. The argument parser defaults this value to ‘pluginify.plugin_packages’, but a user can create separate namespaces if developing interfaces outside of pluginify.

Returns:

error_message – String containing informative error messages from any plugins that were improperly formatted. An exception will be raised at the very end if error_message is not the empty string - this allows collecting ALL errors throughout the plugin registry process and reporting them all at once, to facilitate rapidly identifying and resolving errors.

Return type:

str

pluginify.create_plugin_registries.check_plugin_exists(package, plugins, interface_name, plugin_name, relpath)[source]#

Check if plugin already exists. If it does raise a PluginRegistryError.

Note this only checks for duplicate plugins within a single plugin package. The registry_sanity_check function is used after all plugins have been loaded to identify duplicate plugins across different plugin packages.

Parameters:

package (str) – The package being tested against
plugins (dict) – A dictionary object of all installed plugins in the current plugin package.
interface_name (str) – A string representing the interface being checked against
plugin_name (str) – A string representing the name of the plugin within the interface

Returns:

error_message – Empty string if no error, appropriate informative error message if duplicate plugin found in current plugin package.

Return type:

str

pluginify.create_plugin_registries.collect_class_plugin_metadata(plugin_class)[source]#

Collect metadata linked to the class plugin in ‘module’ for the plugin registry.

Metadata necessary to collect here includes the interface of the plugin, the plugin’s name, its docstring, and the family it adheres to. Other metadata such as relative path and the package it comes from is collected via ‘add_module_plugin’.

Parameters:

plugin_class (Object, default=None) –

The plugin class from ‘module’ to collect metadata from. If None, this function will attempt to locate the correct plugin class from the provided module.

Returns:

metadata –

A dictionary of metadata for the provided plugin class to store in the plugin registry.

Return type:

dict

pluginify.create_plugin_registries.collect_module_plugin_metadata(module, module_name, package, relpath, error_message)[source]#

Collect metadata for the module-based plugin for the plugin registry.

Metadata necessary to collect here includes the interface of the plugin, the plugin’s name, its docstring, and the family it adheres to. Other metadata such as relative path and the package it comes from is collected via ‘add_class_plugin’.

NOTE: The ‘Returns’ section of this docstring is XOR. Either ‘metadata’ will be returned, or ‘error_message’. Not both or neither.

Parameters:

module (ModuleType) –
- The object of the module which has been loaded in realtime. This should have top level attributes such as ‘interface’, ‘name’, ‘family’ and so on.
module_name (str) –
- The name of the .py file.
package (str) –
- The name of the package this plugin comes from
relpath (str) –
- The relative path to the plugin found in ‘package’
error_message (str) –
- The current state of errors found during building the registry, stored as a string.

Returns:

metadata (dict) –
- A dictionary of metadata for the provided plugin class to store in the plugin registry.
error_message (str) –
- The current state of errors found during building the registry, stored as a string.

pluginify.create_plugin_registries.create_plugin_registries(plugin_packages, save_type, namespace)[source]#

Generate all plugin paths associated with every installed packages.

These paths include text plugins, class_based plugins and normal YAML plugins. After these paths are generated, they are sent to parse_plugin_paths, which generates and adds the actual plugins to the plugins dictionary.

Parameters:

plugin_packages (list EntryPoints) – A list of EntryPoints pointing to each installed package –> ie. [EntryPoint(name=’pluginify’, value=’pluginify’, group=’pluginify.plugin_packages’), …]
save_type (str) – The file format to save to [json, yaml]
namespace (str) – Namespace that your plugin packages fall under. The argument parser defaults this value to ‘pluginify.plugin_packages’, but a user can create separate namespaces if developing interfaces outside of pluginify.

pluginify.create_plugin_registries.format_docstring(docstring, use_regex=True)[source]#

Format the provided docstring placement in the plugin registry.

Found when using the CLI and inspecting the registry, some docstrings are formatted in a hard to read manner and look pretty bad. This function will format these docstrings to be easily readable, whether obtained via the CLI or manually inspected in the plugin registry.

Parameters:

docstring (str) –
- The docstring which we are going to format.
use_regex (bool, optional (default=False)) –
- Whether or not we want to apply regex formatting to the docstring. Usually recommended as it will replace ‘newline’ chars but not purposeful ‘.newline’ strings.

pluginify.create_plugin_registries.parse_plugin_paths(plugin_paths, package, package_dir, plugins, namespace)[source]#

Parse the plugin_paths provided from the current installed package.

Then, add them to the plugins dictionary based on the path of the plugin. The path contains information as to whether the plugin is a class_based, text_based, or a yaml_based plugin.

Parameters:

plugin_paths (dict) – A dictionary of filepaths, with keys referring to the type of plugin
package (str) – The current package being parsed
package_dir (str) – The path to the current package (for determining relative paths)
plugins (dict) – A dictionary object of all installed package plugins
namespace (str) – Namespace that your plugin packages fall under. The argument parser defaults this value to ‘pluginify.plugin_packages’, but a user can create separate namespaces if developing interfaces outside of pluginify.

Returns:

error_message – String containing informative error messages from any plugins that were improperly formatted. An exception will be raised at the very end if error_message is not the empty string - this allows collecting ALL errors throughout the plugin registry process and reporting them all at once, to facilitate rapidly identifying and resolving errors.

Return type:

str

pluginify.create_plugin_registries.registry_sanity_check(plugin_packages, save_type, namespace)[source]#

Check that each plugin package registry has no duplicate lowest depth entries.

If it does, raise a PluginRegistryError for that specific package, then remove all plugin registries from each package so the user must fix the error before continuing. While this doesn’t cause a normal error, duplicate plugins will be overwritten by same-named plugin found in the last package-entrypoint.

Parameters:

plugin_packages (list EntryPoints) – A list of EntryPoints pointing to each installed package –> ie. [EntryPoint(name=’pluginify’, value=’pluginify’, group=’pluginify.plugin_packages’)]
save_type (str) – The file format to save to [json, yaml]

Returns:

No returns
Exceptions
———-
PluginRegistryError – If error_message has contents, then raise PluginRegistryError(error_message). The error_message string will collect and report on all errors within this function prior to raising the PluginRegistryError to facilitate rapidly identifying and resolving errors throughout all plugin packages.

pluginify.create_plugin_registries.remove_registries(namespace, plugin_packages)[source]#

Remove all plugin registries if a PluginRegistryError is raised.

Parameters:

namespace (str) – Namespace that your plugin packages fall under. The argument parser defaults this value to ‘pluginify.plugin_packages’, but a user can create separate namespaces if developing interfaces outside of pluginify.
plugin_packages (list[EntryPoints]) – A list of EntryPoints pointing to each installed package –> ie. [EntryPoint(name=’pluginify’, value=’pluginify’, group=’pluginify.plugin_packages’), …]

Return type:

None

pluginify.create_plugin_registries.write_registered_plugins(pkg_dir, plugins, save_type)[source]#

Write dictionary of all plugins available from installed packages.

Parameters:

pkg_dir (str) – Path in which to write registered_plugins
plugins (dict) – A dictionary object of all installed package plugins
save_type (str) – The file format to save to [json, yaml]

Return type:

No returns, file written to pkg_dir

pluginify.errors module#

Pluginify error module.

exception pluginify.errors.PluginError[source]#

Bases: Exception

Exception to be raised when there is an error in a plugin.

exception pluginify.errors.PluginRegistryError[source]#

Bases: Exception

Exception to be raised when there is an error in a plugin registry.

pluginify.plugin_registry module#

PluginRegistry class to interface with the JSON plugin registries.

The “pluginify create” utility generates a JSON file at the top level of every plugin package under a given namespace (default=’pluginify.plugin_packages’) with a complete list of all plugins with the associated metadata (everything except the actual contents of the plugin itself).

Once all of the registered_plugins.json files have been generated via pluginify create, this class uses those registries to quickly identify and open plugins as required. Previously the individual interface classes would open all plugins every time one was required, so moving this process into a single PluginRegistry object allows us to more effectively cache plugins across all interfaces, and avoid reading in all plugins multiple times.

class pluginify.plugin_registry.PluginRegistry(namespace, _test_registry_files=[])[source]#

Bases: object

Plugin Registry class definition.

Represents all of the plugins found in all of the available plugin packages of a given namespace (default=’pluginify.plugin_packages’).

This class will load a plugin when requested, rather than loading all plugins when pluginify is instantiated.

create_registries(packages=None, save_type='json') → None[source]#

Create one or more plugin registry files.

By default, this command will create all plugin registry files for all installed plugin packages under a given namespace (default=pluginify.plugin_packages). If packages is provided via the argument above, create registry files associated with each of those packages.

Parameters:

packages (list[str], default=None) –
- A list of plugin package names corresponding to a namespace whose registries we want to create.
save_type (str, default="json") –
- Format to write registries to. This will also be the file extension. Valid options are either ‘json’ or ‘yaml’.

Raises:

ValueError: –
- Raised if ‘save_type’ is provided but is not one of [‘json’, ‘yaml’]
TypeError: –
- Raised if packages is provided and is not a list of strings.
PluginRegistryError: –
- Raised if one or more of the packages provided is not a valid plugin package under the provided namespace, or if the associated namespace provided is not a valid namespace.

delete_registries(packages=None) → None[source]#

Delete one or more plugin registry files.

By default, this command will delete all plugin registry files found in all installed plugin packages packages (default=pluginify.plugin_packages). If packages is provided via the argument above, delete the registry file(s) associated with each of those packages.

Parameters:

packages (list[str], default=None) –

A list of plugin package names corresponding to a namespace whose registries we want to delete.

Raises:

TypeError: –
- Raised if packages is provided and is not a list of strings.
FileNotFoundError: –
- Raised if a registry file could not be found in one or more plugin packages under self.namespace.
PluginRegistryError: –
- Raised if one or more of the packages provided is not a valid plugin package under the provided namespace, or if the associated namespace provided is not a valid namespace.

get_class_plugin(interface_obj, name, rebuild_registries=None)[source]#

Retrieve a class plugin from this interface by name.

In pluginify’s current state, a class plugin can be a derived plugin object (I.e. legacy module-based plugins) or true class-based plugins which are not derived.

Parameters:

interface_obj (Interface Object) –
- The object representing the interface class requesting this class plugin.
name (str) –
- The name the desired plugin.
rebuild_registries (bool (default=None)) –
- Whether or not to rebuild the registries if get_plugin fails. If set to None, default to True. If specified, use the input value of rebuild_registries, which should be a boolean value. If rebuild registries is true and get_plugin fails, rebuild the plugin registry, call then call get_plugin once more with rebuild_registries toggled off, so it only gets rebuilt once.

Returns:

An object of type <interface>Plugin where <interface> is the name of
this interface.

Raises:

PluginError – If the specified plugin isn’t found within the interface.

get_class_plugins(interface_obj)[source]#

Retrieve all class plugins for this interface.

In pluginify’s current state, this will grab all true class-based plugins and those plugins who were derived into a plugin object from a legacy module-based plugin.

Parameters:

interface_obj (Interface Object) –

The object representing the interface class requesting all plugins.

get_plugin_metadata(interface_obj, plugin_name)[source]#

Retrieve a plugin’s metadata.

Where the metadata of the plugin matches the plugin’s corresponding entry in the plugin registry.

Parameters:

interface_obj (Interface Object) –
- The object representing the interface class requesting plugin metadata.
plugin_name (str or tuple(str)) –
- The name of the plugin whose metadata we want.

Returns:

metadata –

A dictionary of metadata for the requested plugin.

Return type:

dict

get_yaml_plugin(interface_obj, name, rebuild_registries=None, _expand=False)[source]#

Get a YAML plugin by its name.

Parameters:

interface_obj (Interface Object) –
- The object representing the interface class requesting this yaml plugin.
name (str or tuple(str)) –
- The name of the yaml-based plugin. Either a single string or a tuple of strings for product plugins.
rebuild_registries (bool (default=None)) –
- Whether or not to rebuild the registries if get_plugin fails. If set to None, default to True. If specified, use the input value of rebuild_registries, which should be a boolean value. If rebuild registries is true and get_plugin fails, rebuild the plugin registry, call then call get_plugin once more with rebuild_registries toggled off, so it only gets rebuilt once.
_expand (private bool (default=False)) –
- If true, fully expand the workflow plugin in place. Otherwise, load as is done usually. This should only be used for the ‘geoips expand <workflow>’ command.

get_yaml_plugins(interface_obj)[source]#

Retrieve all yaml plugin objects for this interface.

Parameters:

interface_obj (Interface Object) –

The object representing the interface class requesting all plugins.

property interface_mapping#

Dictionary of interface types and interfaces of that type.

pluginify has three types of interfaces, though only two are commonly used (yaml_based, module_based). This dictionary has top level keys of all interface types, with their values being the list of unique interfaces that inherit that type.

load_plugin(data: dict, _expand: bool = False) → BaseModel[source]#

Dynamically load and validate pydantic models based on apiVersion and interface.

This method parses the apiVersion field from the input dictionary to determine the package and pydantic model version to use. It then dynamically imports the corresponding Pydantic model classes based on the interface field to validate the input data.

Parameters:

data (dict) – Dictionary representing a plugin definition. Must include the interface field. May optionally include apiVersion. If not present, “pluginify/v1” is assumed.
_expand (private bool (default=False)) – If true, fully expand the workflow plugin in place. Otherwise, load as is done usually. This should only be used for the ‘geoips expand <workflow>’ command.

Returns:

A validated Pydantic model instance.

Return type:

BaseModel

Raises:

ValueError – If apiVersion is improperly formatted or if interface field is missing.
ImportError – If the specified module for the given model version cannot be imported.

property registered_class_based_plugins#: A dictionary of registered class-based plugins.

property registered_plugins#

Dictionary of every plugin’s metadata found within self.namespace.

self.namespace usually should correspond to ‘pluginify.plugin_packages’ unless you’re creating registries for plugins that exist outside this namespace.

property registered_yaml_based_plugins#: A dictionary of registered YAML-based plugins.

retry_get_plugin(interface_obj, name, rebuild_registries, err_str, err_type=<class 'pluginify.errors.PluginError'>)[source]#

Rerun self.get_plugin, but call ‘pluginify create’ beforehand.

By running ‘pluginify create’, we automate the registration of plugins under a given namespace (default=’pluginify.plugin_packages’). If the plugin persists not to be found, then we’ll raise an appropriate PluginError as denoted by ‘err_str’.

Parameters:

interface_obj (Interface Object) –
- The object representing the interface class requesting this plugin.
name (str or tuple(str)) –
- The name of the yaml plugin. Either a single string or a tuple of strings for product plugins.
rebuild_registries (bool) –
- Whether or not to rebuild the registries if get_plugin fails. If set to true and get_plugin fails, rebuild the plugin registry, call then call get_plugin once more with rebuild_registries toggled off, so it only gets rebuilt once.
err_str (string) –
- The error to be reported.
err_type (Exception-based Class) –
- The class of exception to be raised.

pluginify.plugin_registry.env_constructor(loader, node)[source]#

YAML constructor for the !ENV tag that expands environment variables.

This function allows YAML files to include environment variables using the !ENV tag. The scalar value associated with the tag is read and any environment variables within the string (e.g. ${VAR_NAME}) are expanded using os.path.expandvars().

Parameters:

loader (yaml.Loader) – The YAML loader instance currently parsing the document.
node (yaml.Node) – The YAML node containing the scalar value associated with the !ENV tag.

Returns:

The scalar value with any environment variables expanded using the current process environment.

Return type:

str

Module contents#

Pluginify __init__ module.