Data Passing with Volumes¶
This guide will walk you through understanding how Kale uses snapshots and volumes for data passing.
What You’ll Need¶
- An EKF or MiniKF deployment with the default Kale Docker image.
- An understanding of how Kale SDK works.
Create a new Notebook server using the default Kale Docker image. The image will have the following naming scheme:gcr.io/arrikto/jupyter-kale-py38:<IMAGE_TAG>
<IMAGE_TAG>varies based on the MiniKF or EKF release.
Connect to the server, open a terminal, and install
scikit-learn:$ pip3 install --user scikit-learn==0.23.0
Create a new python file and name it
kale_data_passing.py:$ touch kale_data_passing.py
Copy and paste the following code inside
1 # Copyright © 2021-2022 Arrikto Inc. All Rights Reserved. 2 3 """Kale SDK. 4 5 This script trains an ML pipeline to solve a binary classification task. 6 """ 7 8 from kale.sdk import pipeline, step 9 from sklearn.datasets import make_classification 10 from sklearn.linear_model import LogisticRegression 11 from sklearn.model_selection import train_test_split 12 13 14 @step(name="data_loading") 15 def load(random_state): 16 """Create a random dataset for binary classification.""" 17 rs = int(random_state) 18 x, y = make_classification(random_state=rs) 19 return x, y 20 21 22 @step(name="data_split") 23 def split(x, y): 24 """Split the data into train and test sets.""" 25 x, x_test, y, y_test = train_test_split(x, y, test_size=0.1) 26 return x, x_test, y, y_test 27 28 29 @step(name="model_training") 30 def train(x, x_test, y, training_iterations): 31 """Train a Logistic Regression model.""" 32 iters = int(training_iterations) 33 model = LogisticRegression(max_iter=iters) 34 model.fit(x, y) 35 print(model.predict(x_test)) 36 37 38 @pipeline(name="binary-classification", experiment="kale-tutorial") 39 def ml_pipeline(rs=42, iters=100): 40 """Run the ML pipeline.""" 41 x, y = load(rs) 42 x, x_test, y, y_test = split(x=x, y=y) 43 train(x, x_test, y, training_iterations=iters) 44 45 46 if __name__ == "__main__": 47 ml_pipeline(rs=42, iters=100)
In this code sample, you start with a standard Python script that trains a Logistic Regression model. Moreover, you have decorated the functions using the Kale SDK. To read more about how to create this file, head to the corresponding Kale SDK user guide.
Compile and run the pipeline, note that Kale takes a snapshot of the Notebook’s volumes:$ python3 -m kale kale_parameters.py --kfp [...] 2021-09-21 09:07:50 Kale rokutils:281 [INFO] Creating Rok bucket 'notebooks'... 2021-09-21 09:07:50 Kale rokutils:291 [INFO] Rok bucket 'notebooks' already exists 2021-09-21 09:07:51 Kale rokutils:157 [INFO] Monitoring Rok snapshot with task id: 1556b315b8b54aeaa6a5a241cf75e42a Rok Task: |********************************| 100/100 [...]
Navigate to the KFP UI to the run page. Note that the new pipeline includes a step called
create-volume-1. This step is responsible for provisioning a clone of the Notebook’s workspace volume. If the Notebook had also some data volumes, we would see more pipeline steps called
create-volume-*, each one provisioning a clone of a data volume.
Volumes and Snapshots¶
When you create your Notebook Server using the Jupyter Web App, by default a new PVC is mounted under the user’s home directory. We call this volume, workspace volume. Moreover, you can decide to provision more volumes to be mounted at locations of your choice. We call these volumes, data volumes.
Whenever you submit a pipeline to KFP, Kale clones the Notebook’s volumes for two important reasons:
Marshalling: a mechanism to seamlessly pass data between steps. This system requires a shared folder where Kale can serialize and de-serialize data.
Kale uses a hidden folder within the workspace volume as the shared marshalling location.
Reproducibility, experimentation: when you are working on a Notebook, it is often the case that you install new libraries, write new modules, create or download assets required by your code. By seamlessly cloning the workspace and data volumes, all your environment is versioned and replicated to the new pipeline. Thus, the pipeline is always reproducible, thanks to the immutable snapshots, and you do not have to build new Docker images for each pipeline run.
Currently Kale requires to take a snapshot of the Notebook volumes, because every step needs to be sourced from its original Python module or Notebook. In the near future we will provide an option to compile and run pipelines without needing to take volume snapshots, and at the same time ensure the immutability and reproducibility of the pipeline steps.
Using the workspace volume to marshal data can become problematic in case your
pipeline steps need to pass large assets between them. Imagine if the
step of the pipeline above produced a dataset too large for the workspace
volume to contain it. You can ask Kale to provision a volume dedicated to data
passing, deciding its size and mount location:
|1||# Copyright © 2021-2022 Arrikto Inc. All Rights Reserved.|
|5||This script trains an ML pipeline to solve a binary classification task.|
|8||from kale.sdk import pipeline, step|
|9||from sklearn.datasets import make_classification|
|10||from sklearn.linear_model import LogisticRegression|
|11||from sklearn.model_selection import train_test_split|
|16||"""Create a random dataset for binary classification."""|
|17||rs = int(random_state)|
|18||x, y = make_classification(random_state=rs)|
|19||return x, y|
|23||def split(x, y):|
|24||"""Split the data into train and test sets."""|
|25||x, x_test, y, y_test = train_test_split(x, y, test_size=0.1)|
|26||return x, x_test, y, y_test|
|30||def train(x, x_test, y, training_iterations):|
|31||"""Train a Logistic Regression model."""|
|32||iters = int(training_iterations)|
|33||model = LogisticRegression(max_iter=iters)|
|42||def ml_pipeline(rs=42, iters=100):|
|43||"""Run the ML pipeline."""|
|44||x, y = load(rs)|
|45||x, x_test, y, y_test = split(x=x, y=y)|
|46||train(x, x_test, y, training_iterations=iters)|
|49||if __name__ == "__main__":|
When you activate the marshal volume, Kale uses it to marshal data between steps, but still keeps the Notebook’s clones, for the reasons explained above:
You can also use the marshal volume to perform data passing yourself, in case you need to write files to a shared location and have downstream steps consume them.
You have learned how Kale uses volumes and snapshots to perform data passing and how to create a marshal volume.
The next step is to learn how to use type-hints for the step parameters of your pipelines.