Configure Argo Workflow Executor

A workflow executor allows Argo to perform certain actions like monitoring Pod logs, collecting artifacts, managing the life cycle of containers, etc. Certain executors introduce some limitations in specific scenarios or environments, so the admin might need to change the executor depending on their needs.

For example, here are some known issues with certain executors:

• PNS executor: Kubeflow Pipelines fail when Pods are spawned in a newly created node or when they involve short-lived containers.
• Docker executor: Kubeflow Pipelines do not work on AKS clusters with Linux node pools running Kubernetes 1.19 or newer, since they no longer support Docker.

See also

• Official documentation for Argo workflow executors.
• Official AKS documentation on container runtime configuration.

This guide will walk you through configuring a custom Argo workflow executor. Here is the suggested Argo workflow executor you can use for Kubeflow Pipelines, based on your cloud provider:

AWS

Azure

Google Cloud

If your Kubernetes cluster is 1.22 or newer, the Docker container runtime is not supported, so use the PNS executor. Otherwise, use the Docker executor.

If your Kubernetes cluster is 1.19 or newer, the Docker container runtime is not supported, so use the PNS executor. Otherwise, use the Docker executor.

If your Kubernetes cluster is 1.24 or newer or uses nodes that do not have an UBUNTU image type, the Docker container runtime is not supported, so use the PNS executor. Otherwise, use the Docker executor.

To ensure that your cluster supports Docker, inspect the image type of your nodes:

rok-tools:~# gcloud container clusters describe ${GKE_CLUSTER?} | \ > grep imageType imageType: UBUNTU imageType: UBUNTU

Important

Ensure that all image types are UBUNTU. If even one image type has a different value, then this node pool does not support Docker and so, you cannot use the Docker executor.

What You’ll Need

• A configured management environment.
• An Arrikto EKF deployment.

Procedure

1. Go to your GitOps repository, inside your rok-tools management environment:

   

   root@rok-tools:~# cd ~/ops/deployments

2. Create a patches directory in the Kubeflow Pipelines deploy overlay:

   

   root@rok-tools:~/ops/deployments# mkdir -p kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches

3. Create the kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml patch:

   

   root@rok-tools:~/ops/deployments# touch kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml

4. Edit the kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml patch to configure the executor that the Argo workflow controller will use. Append the following lines if the file is empty or edit the existing patch accordingly if it is not. Choose one of the following options, based on the executor you will use.

   Docker

   PNS

   

   apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap data: containerRuntimeExecutor: docker

   

   apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap data: containerRuntimeExecutor: pns

5. Edit the kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/kustomization.yaml file and append the following lines to enable the previous patch. Edit the existing patches directive instead, if it is already present:

   

   patches: - patches/workflow-controller-configmap.yaml

6. Stage your changes:

   

   root@rok-tools:~/ops/deployments# git add kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml

7. Commit your changes:

   

   root@rok-tools:~/ops/deployments# git commit -am "Configure Argo workflow executor"

8. Apply the Kubeflow Pipelines manifests:

   

   root@rok-tools:~/ops/deployments# rok-deploy --apply kubeflow/manifests/apps/pipeline/upstream/overlays/deploy

Verify

1. Verify that the ConfigMap of the Argo workflow controller has been updated with the executor you specified earlier:

   Docker

   PNS

   

   root@rok-tools:~# kubectl get -n kubeflow configmap workflow-controller-configmap -o json | \ > jq -r '.data.containerRuntimeExecutor' docker

   

   root@rok-tools:~# kubectl get -n kubeflow configmap workflow-controller-configmap -o json | \ > jq -r '.data.containerRuntimeExecutor' pns

Summary

You have successfully configured the Argo workflow executor.

What’s Next

Check out the rest of the operations you can perform on your Kubeflow deployment.

• Manage Your Kubeflow Deployment

PreviousNext