Access Kubeflow Pipelines from External Client

This guide contains instructions to access Kubeflow Pipelines (KFP) in your Kubeflow deployment from an external client.

The only way to access Kubeflow Pipelines is by using the KFP Software Development Kit (SDK).

Note

We do not currently support accessing KFP from external clients through the KFP CLI.

What You’ll Need

Contact Arrikto

If you are using a custom image or environment, make sure it supports Python3 and contact the Arrikto Tech Team to provide you with the wheels to install Arrikto’s KFP SDK fork.

Procedure

Choose one of the following options, based on whether you will use a terminal or a Python3 Jupyter notebook.

  1. Retrieve the version of the Arrikto fork of the KFP client that you are using:

    jovyan@mynotebook-0:~$ pip3 list | grep kfp kfp 1.7.1 ...

    The previous output shows that the version of KFP we are using is 1.7.1. For the next steps, choose one of the following options based on your version.

  2. Export the token path in an environment variable. Replace <PATH_TO_TOKEN> with the path to your token file.

    jovyan@mynotebook-0:~$ export KF_PIPELINES_SA_TOKEN_PATH=<PATH_TO_TOKEN>
    jovyan@mynotebook-0:~$ export ML_PIPELINE_SA_TOKEN_PATH=<PATH_TO_TOKEN>
  3. Start a Python kernel:

    jovyan@mynotebook-0:~$ python3
  4. Specify the host:

    # Set the host to the Kubeflow cluster URL host = <HOST_URL>

    Replace <HOST_URL> with the URL of the Kubeflow cluster, for example:

    # Set the host to the Kubeflow cluster URL host = "https://my-cluster.aws.com"
  5. Set up the credentials and initialize a KFP client:

    import kfp kubernetes_url = "%s/kubernetes" % host pipelines_url = "%s/pipeline" % host creds = kfp.auth.ServiceAccountTokenCredentials(kubernetes_url=kubernetes_url) client = kfp.Client(host=pipelines_url, credentials=creds)
    import kfp kubernetes_url = "%s/kubernetes" % host pipelines_url = "%s/pipeline" % host creds = kfp.ServiceAccountTokenCredentials(kubernetes_url=kubernetes_url) client = kfp.Client(host=pipelines_url, credentials=creds)
  6. Select a namespace to perform requests against:

    client.set_user_namespace(<NAMESPACE>)

    Replace <NAMESPACE> with the desired namespace, for example:

    client.set_user_namespace("kubeflow-user")
  7. Use the client to perform all the CRUD operations supported by the REST API. For example:

    client.list_experiments() client.create_experiment(name="test") client.get_experiment(experiment_id="<id>") client.list_pipelines() client.run_pipeline(experiment_id="<id>", pipeline_id="<id>") client.list_runs() client.list_recurring_runs()
  1. Retrieve the version of the Arrikto fork of the KFP client that you are using:

    In [1]: import kfp In [2]: kfp.__version__ Out[2]: '1.7.1'

    The previous output shows that the version of KFP we are using is 1.7.1. For the next steps, choose one of the following options based on your version.

  2. Export the token path in an environment variable. Replace <PATH_TO_TOKEN> with the path to your token file:

    import os os.environ["KF_PIPELINES_SA_TOKEN_PATH"] = <PATH_TO_TOKEN>
    import os os.environ["ML_PIPELINE_SA_TOKEN_PATH"] = <PATH_TO_TOKEN>
  3. Specify the host:

    # Set the host to the Kubeflow cluster URL host = <HOST_URL>

    Replace <HOST_URL> with the URL of the Kubeflow cluster, for example:

    # Set the host to the Kubeflow cluster URL host = "https://my-cluster.aws.com"
  4. Set up the credentials and initialize a KFP client:

    import kfp kubernetes_url = "%s/kubernetes" % host pipelines_url = "%s/pipeline" % host creds = kfp.auth.ServiceAccountTokenCredentials(kubernetes_url=kubernetes_url) client = kfp.Client(host=pipelines_url, credentials=creds)
    import kfp kubernetes_url = "%s/kubernetes" % host pipelines_url = "%s/pipeline" % host creds = kfp.ServiceAccountTokenCredentials(kubernetes_url=kubernetes_url) client = kfp.Client(host=pipelines_url, credentials=creds)
  5. Select a namespace to perform requests against:

    client.set_user_namespace(<NAMESPACE>)

    Replace <NAMESPACE> with the desired namespace, for example:

    client.set_user_namespace("kubeflow-user")
  6. Use the client to perform all the CRUD operations supported by the REST API. For example:

    client.list_experiments() client.create_experiment(name="test") client.get_experiment(experiment_id="<id>") client.list_pipelines() client.run_pipeline(experiment_id="<id>", pipeline_id="<id>") client.list_runs() client.list_recurring_runs()

See also

  • Find all the functions that Client implements by navigating to the source repository.

Summary

You have successfully accessed KFP using an external client.

What’s Next

Check out the rest of the documentation regarding KFP.