Access Kubeflow Pipelines¶

There are two ways to access Kubeflow Pipelines:

Using the KFP Software Development Kit (SDK)
Using the KFP Command Line Interface (CLI)

Note

The KFP CLI method is currently only supported when accessing from a pod

Arrikto's Kubeflow cluster comes with several security features that are not present in the open-source distribution. For this reason, remember to always install the Arrikto distribution of the Kubeflow Pipelines SDK and Client in order to properly authenticate to the KFP apiserver.

Software Development Kit (SDK)¶

To interact with the apiserver programmatically, you will need to use the KFP Python SDK in your application.

Access from an external client¶

Make sure you have first followed the instructions described in the Set up an external client section. You should end up with a file containing your identity's token.

Retrieve the version of the Arrikto fork of the KFP client that you are using:
```
$ pip3 list | grep kfp
kfp                           1.7.1
...
```
The previous output shows that the version of KFP we are using is 1.7.1. For the next steps, choose one of the following options based on your version.

Export the file path in an environment variable:

$ export KF_PIPELINES_SA_TOKEN_PATH="/path/to/token"

$ export ML_PIPELINE_SA_TOKEN_PATH="/path/to/token"

Start a Python kernel and specify the host:

# Set the host to the Kubeflow cluster URL
host = "https://my-cluster.aws.com"

Set up the credentials and initialize a KFP client:

import kfp
kubernetes_url = "%s/kubernetes" % host
pipelines_url = "%s/pipeline" % host
creds = kfp.auth.ServiceAccountTokenCredentials(kubernetes_url=kubernetes_url)
client = kfp.Client(host=pipelines_url, credentials=creds)

import kfp
kubernetes_url = "%s/kubernetes" % host
pipelines_url = "%s/pipeline" % host
creds = kfp.ServiceAccountTokenCredentials(kubernetes_url=kubernetes_url)
client = kfp.Client(host=pipelines_url, credentials=creds)

Select a namespace to perform requests against:
```
client.set_user_namespace("kubeflow-user")
```

Access from a pod¶

Create a Notebook Server. Make sure you have enabled the configuration allowing access to Kubeflow Pipelines.

Note

To have access to Kubeflow Pipelines, a pod needs to have the corresponding PodDefault enabled.

Start a Python kernel and create a new KFP Client:

import kfp
client = kfp.Client()
# When in-cluster, the client automatically sets the requests' namespace
# to the pod's namespace

Now you can use the client to perform all the CRUD operations supported by the REST API. Some examples:

client.list_experiments()
client.create_experiment(name="test")
client.get_experiment(experiment_id="<id>")

client.list_pipelnes()
client.run_pipeline(experiment_id="<id>", pipeline_id="<id>")

client.list_runs()
client.list_recurring_runs()

You can see all the functions that Client implements by navigating to the source repository.

Interactive Command Line Interface (CLI)¶

Access from an external client¶

Important

We do not currently support this method.

Access from a pod¶

Create a Notebook Server. Make sure you have enabled the configuration allowing access to Kubeflow Pipelines.

You can interact with the KFP apiserver using the kfp CLI command as shown in the public KFP documentation.

The CLI utility allows you to interact with the pipeline/experiment/run CRUD APIs.

Run kfp --help / kfp <command> --help (e.g., kfp pipeline --help) for more information.