Pipeline Parameter Macros

Kubeflow Pipelines provides pipeline parameter macros to be used in recurring runs. These macros are useful in case your recurring pipeline needs to perform some task that is contextual to the current run time.

Arrikto has this upstream functionality to include more macros and make them available in one-off runs, as well.

How to apply a macro

To apply a macro to a pipeline run, you first need to parameterize it. This means that the pipeline accepts input arguments (pipeline parameters). If you are using the Kale SDK to create the pipeline, check out the corresponding Kale SDK guide to learn how to build a parameterized pipeline.

To use a macro in one of your pipeline parameters, just set the value of the parameter to [[<macro-name>]]. The Kubeflow Pipelines apiserver will inspect the input parameters and substitute the macros accordingly, just before submitting the pipeline run.

One-off runs

This is a list of macros available in one-off runs:

  1. [[RunUUID]]: Replaced with the unique run ID. This unique identifier is the ID that KFP uses internally to store and reference that specific run. You can use this information, for example, to leave an auditable trace when your pipeline interacts with some external service.
  2. [[CurrentTime]]: Replaced with a timestamp corresponding to the KFP run submission time. This information can be useful in case your code depends on the current time, or if you want to test a pipeline that will run as a scheduled job.

Recurring runs

This is a list of macros available in recurring runs:

  1. [[RunUUID]]: See previous section.
  2. [[CurrentTime]]: See previous section.
  3. [[ScheduledTime]]: Replaced with the scheduled time of the workflow. This is useful because the run will never start at the exact time it was supposed to. You can expect [[CurrentTime]] to provide a timestamp that differs from [[ScheduledTime]] by a few seconds.
  4. [[Index]]: Replaced by the index of the current run. For instance, if the run is the 6th being run as part of a recurring job, this macro will be replaced with the value 6.

Timestamp formatting

Macros that provide a timestamp as output (CurrentTime and ScheduledTime) can be formatted using GoLang’s date format system.

Formatting a timestamp in Go requires providing a reference datetime in the expected layout. The reference date is Mon Jan 2 15:04:05 -0700 MST 2006. Take a look at the examples in this guide for a comparison between the Go layout and the more common Java layout.

To apply a timestamp template to the macros, use the following syntax: [[ScheduledTime.<template>]] (e.g., [[ScheduledTime.2006-01-02]]).