The `workflows` interface#

A workflow is a YAML plugin (interface: workflows, family: order_based) that defines an ordered sequence of OBP steps. This page is the reference for how a workflow is structured.

Note

If you have not read Order-Based Processing (OBP) yet, start there — this page assumes the ETCVO model and the basic step vocabulary.

Top-level fields#

A workflow plugin uses the standard GeoIPS plugin header plus a spec:

apiVersion: geoips/v1
interface: workflows
family: order_based
name: my_workflow            # unique, valid Python identifier
docstring: What this workflow produces.
package: geoips
spec:
  globals: {}                # arguments applied to every step (optional)
  steps: {}                  # the ordered steps (required)

Steps#

spec.steps maps a step id to a step definition. The step id is a unique, valid Python identifier and also names the step’s output in the processing tree.

Each step definition accepts:

kind: The plugin interface for this step (reader, algorithm, interpolator, colormapper, sector, coverage_checker, filename_formatter, output_formatter, output_checker, product, product_default, or workflow).
name: The specific plugin of that kind (for product steps, a [source, product] pair).
arguments: A mapping validated against the plugin’s call signature. Unsupported arguments are rejected by Pydantic before processing starts.
depends_on: A list of step ids whose output feeds this step. If omitted, GeoIPS infers the most recent data-producing step. Use dotted notation to depend on a step inside an embedded workflow: depends_on: [abi_Infrared.algorithm].
keep: Whether to retain this step’s result in the processing tree after it is consumed (defaults to a retention policy — see Retention policies).

Steps must accept data conforming to the GeoIPS xarray standard (except reader steps, which produce it) and return data in the same standard (except output_formatter steps, which write files).

Global and kind arguments#

spec.globals supplies arguments applied to every step (e.g. sector_list, logging_level, product_name). Arguments a given plugin does not accept are dropped automatically before that plugin is called. A workflow’s test section can also carry a kinds block that applies arguments to every step of a given interface kind (e.g. a satellite_zenith_angle_cutoff for all readers).

Embedded workflows (reuse)#

A step of kind: workflow, kind: product, or kind: product_default expands into an embedded set of steps at runtime, so you can compose complex workflows from smaller ones:

Workflow-in-workflow — reference another registered workflow by name; its steps are loaded and nested in place.
Product-in-workflow — a product step is expanded into the ordered steps implied by its product family (e.g. interpolator_algorithm_colormapper inserts an interpolator, then an algorithm, then a colormapper).
Product-default-in-workflow — same idea, driven by a product_default.

When a product/product_default expands, the first data-consuming step depends on the last data output from the parent workflow, and subsequent steps chain from there. See Running workflows for a fully worked expansion example.

The ordered product families (about 95% of GeoIPS products) are:

ORDERED_PRODUCT_FAMILIES = [
    "algorithm",
    "algorithm_colormapper",
    "algorithm_interpolator_colormapper",
    "interpolator",
    "interpolator_algorithm",
    "interpolator_algorithm_colormapper",
]

The `test` section#

A workflow can carry a test section describing how it is validated in CI: the input fnames, expected outputs (with comparison paths and output checkers), and override blocks (steps, kinds, globals). The same override structure can be supplied on the command line — see Running workflows.

test:
  fnames: !ENV ${GEOIPS_TESTDATA_DIR}/test_data_abi/data/goes16_20200918_1950/*
  outputs:
    output_formatter:
      full_test_policy: on_token_mismatch   # always | never
      compare_path: !ENV $GEOIPS_PACKAGES_DIR/geoips/tests/outputs/abi.static.<product>.imagery_clean
      output_checker_name: image

Model reference#

The workflow YAML is validated against the following Pydantic model. The fields below are generated directly from the code, so they always match the current validation rules:

pydantic model geoips.pydantic_models.v1.workflows.WorkflowPluginModel[source]

A plugin that produces a workflow.

Fields:

Validators:

propagate_context » all fields

field spec: WorkflowSpecModel [Required]: The workflow specification

field test: WorkflowTestModel = None: An optional dictionary of parameters used to test this workflow.

The workflows interface#