Configure Argo Workflow Executor

A workflow executor allows Argo to perform certain actions like monitoring Pod logs, collecting artifacts, managing the life cycle of containers, etc. Certain executors introduce some limitations in specific scenarios or environments, so the admin might need to change the executor depending on their needs.

For example, here are some known issues with certain executors:

  • PNS executor: Kubeflow Pipelines fail when Pods are spawned in a newly created node or when they involve short-lived containers.
  • Docker executor: Kubeflow Pipelines do not work on AKS clusters with Linux node pools running Kubernetes 1.19 or newer, since they no longer support Docker.
  • Emissary executor: Kubeflow Pipelines with custom steps will fail if the user does not explicitly define a command and args for the container.

This guide will walk you through configuring a custom Argo workflow executor.

Note

We suggest using the Emissary Argo workflow executor, which is the default executor for Kubeflow Pipelines, unless you want to specify custom KFP steps without explicitly specifying command and args for the containers.

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Create a patches directory in the Kubeflow Pipelines deploy overlay:

    root@rok-tools:~/ops/deployments# mkdir -p kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches
  3. Create the kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml patch:

    root@rok-tools:~/ops/deployments# touch kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml
  4. Edit the kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml patch to configure the executor that the Argo workflow controller will use. Append the following lines if the file is empty or edit the existing patch accordingly if it is not. Choose one of the following options, based on the executor you will use.

    apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap data: containerRuntimeExecutor: docker
    apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap data: containerRuntimeExecutor: pns
    apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap data: containerRuntimeExecutor: emissary
  5. Edit the kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/kustomization.yaml file and append the following lines to enable the previous patch. Edit the existing patches directive instead, if it is already present:

    patches: - patches/workflow-controller-configmap.yaml
  6. Stage your changes:

    root@rok-tools:~/ops/deployments# git add kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml
  7. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit -am "Configure Argo workflow executor"
  8. Apply the Kubeflow Pipelines manifests:

    root@rok-tools:~/ops/deployments# rok-deploy --apply kubeflow/manifests/apps/pipeline/upstream/overlays/deploy

Verify

  1. Verify that the ConfigMap of the Argo workflow controller has been updated with the executor you specified earlier:

    root@rok-tools:~# kubectl get -n kubeflow configmap workflow-controller-configmap -o json \ > | jq -r '.data.containerRuntimeExecutor' docker
    root@rok-tools:~# kubectl get -n kubeflow configmap workflow-controller-configmap -o json \ > | jq -r '.data.containerRuntimeExecutor' pns
    root@rok-tools:~# kubectl get -n kubeflow configmap workflow-controller-configmap -o json \ > | jq -r '.data.containerRuntimeExecutor' emissary

Summary

You have successfully configured the Argo workflow executor.

What’s Next

Check out the rest of the operations you can perform on your Kubeflow deployment.