Access Kubeflow Pipelines¶
There are two ways to access Kubeflow Pipelines:
- Using the KFP Software Development Kit (SDK)
- Using the KFP Command Line Interface (CLI)
Note
The KFP CLI method is currently only supported when accessing from a pod
Arrikto's Kubeflow cluster comes with several security features that are not present in the open-source distribution. For this reason, remember to always install the Arrikto distribution of the Kubeflow Pipelines SDK and Client in order to properly authenticate to the KFP apiserver.
Software Development Kit (SDK)¶
To interact with the apiserver programmatically, you will need to use the KFP Python SDK in your application.
Access from an external client¶
Make sure you have first followed the instructions described in the Set up an external client section. You should end up with a file containing your identity's token.
Retrieve the version of the Arrikto fork of the KFP client that you are using:
$ pip3 list | grep kfp kfp 1.7.1 ...
The previous output shows that the version of KFP we are using is
1.7.1
. For the next steps, choose one of the following options based on your version.Export the file path in an environment variable:
$ export KF_PIPELINES_SA_TOKEN_PATH="/path/to/token"
$ export ML_PIPELINE_SA_TOKEN_PATH="/path/to/token"
Start a Python kernel and specify the host:
# Set the host to the Kubeflow cluster URL host = "https://my-cluster.aws.com"
Set up the credentials and initialize a KFP client:
import kfp kubernetes_url = "%s/kubernetes" % host pipelines_url = "%s/pipeline" % host creds = kfp.auth.ServiceAccountTokenCredentials(kubernetes_url=kubernetes_url) client = kfp.Client(host=pipelines_url, credentials=creds)
import kfp kubernetes_url = "%s/kubernetes" % host pipelines_url = "%s/pipeline" % host creds = kfp.ServiceAccountTokenCredentials(kubernetes_url=kubernetes_url) client = kfp.Client(host=pipelines_url, credentials=creds)
Select a namespace to perform requests against:
client.set_user_namespace("kubeflow-user")
Access from a pod¶
Create a Notebook Server. Make sure you have enabled the configuration allowing access to Kubeflow Pipelines.
Note
To have access to Kubeflow Pipelines, a pod needs to have the corresponding
PodDefault
enabled.Start a Python kernel and create a new KFP Client:
import kfp client = kfp.Client() # When in-cluster, the client automatically sets the requests' namespace # to the pod's namespace
Now you can use the client to perform all the CRUD operations supported by the REST API. Some examples:
client.list_experiments()
client.create_experiment(name="test")
client.get_experiment(experiment_id="<id>")
client.list_pipelnes()
client.run_pipeline(experiment_id="<id>", pipeline_id="<id>")
client.list_runs()
client.list_recurring_runs()
You can see all the functions that Client
implements by navigating to the
source repository.
Interactive Command Line Interface (CLI)¶
Access from an external client¶
Important
We do not currently support this method.
Access from a pod¶
Create a Notebook Server. Make sure you have enabled the configuration allowing access to Kubeflow Pipelines.
You can interact with the KFP apiserver using the kfp
CLI command as shown
in the public KFP documentation.
The CLI utility allows you to interact with the
pipeline
/experiment
/run
CRUD APIs.
Run kfp --help
/ kfp <command> --help
(e.g., kfp pipeline --help
)
for more information.