Pipeline Parameter Macros¶
Kubeflow Pipelines provides pipeline parameter macros to be used in recurring runs. These macros are useful in case your recurring pipeline needs to perform some task that is contextual to the current run time.
Arrikto has this upstream functionality to include more macros and make them available in one-off runs, as well.
How to apply a macro¶
To apply a macro to a pipeline run, you first need to parameterize it. This means that the pipeline accepts input arguments (pipeline parameters). If you are using the Kale SDK to create the pipeline, check out the corresponding Kale SDK guide to learn how to build a parameterized pipeline.
To use a macro in one of your pipeline parameters, just set the value of the
parameter to [[<macro-name>]]
. The Kubeflow Pipelines apiserver will
inspect the input parameters and substitute the macros accordingly, just before
submitting the pipeline run.
One-off runs¶
This is a list of macros available in one-off runs:
[[RunUUID]]
: Replaced with the unique run ID. This unique identifier is the ID that KFP uses internally to store and reference that specific run. You can use this information, for example, to leave an auditable trace when your pipeline interacts with some external service.[[CurrentTime]]
: Replaced with a timestamp corresponding to the KFP run submission time. This information can be useful in case your code depends on the current time, or if you want to test a pipeline that will run as a scheduled job.
Recurring runs¶
This is a list of macros available in recurring runs:
[[RunUUID]]
: See previous section.[[CurrentTime]]
: See previous section.[[ScheduledTime]]
: Replaced with the scheduled time of the workflow. This is useful because the run will never start at the exact time it was supposed to. You can expect[[CurrentTime]]
to provide a timestamp that differs from[[ScheduledTime]]
by a few seconds.[[Index]]
: Replaced by the index of the current run. For instance, if the run is the 6th being run as part of a recurring job, this macro will be replaced with the value6
.
Timestamp formatting¶
You can format macros that provide a timestamp as output (CurrentTime
and
ScheduledTime
) using GoLang’s date format system.
Formatting a timestamp in Go requires you to provide a reference datetime in
the expected layout. The reference date is Mon Jan 2 15:04:05 -0700 MST
2006
. Take a look at the examples in this guide for
a comparison between the Go layout and the more common Java layout.
To apply a timestamp template to the macros, use the following syntax:
[[ScheduledTime.<template>]]
(e.g., [[ScheduledTime.2006-01-02]]
).