Create KFP HTML Artifacts

You can use Kale to create KFP HTML artifacts, which you can use to provide rich performance evaluation metrics and figures. This guide will walk you through creating KFP HTML artifacts, to visualize through the KFP UI, using the Kale SDK.

What You’ll Need

  • An EKF or MiniKF deployment with the default Kale Docker image.
  • An understanding of how Kale SDK works.

Procedure

  1. Create a new Notebook server using the default Kale Docker image. The image will have the following naming scheme:

    gcr.io/arrikto/jupyter-kale-py36:<IMAGE_TAG>
    

    Note

    The <IMAGE_TAG> varies based on the MiniKF or EKF release.

  2. Connect to the server, open a terminal, and install scikit-learn and matplotlib:

    $ pip3 install --user scikit-learn==0.23.0 matplotlib==3.3.0
    
  3. Create a new python file and name it kale_artifacts.py:

    $ touch kale_artifacts.py
    
  4. Copy and paste the following code inside kale_artifacts.py:

    # Copyright © 2021 Arrikto Inc.  All Rights Reserved.
    
    """Kale SDK.
    
    This script trains an ML pipeline to solve a binary classification task.
    """
    
    from kale.sdk import pipeline, step
    from sklearn.datasets import make_classification
    from sklearn.linear_model import LogisticRegression
    from sklearn.model_selection import train_test_split
    
    
    @step(name="data_loading")
    def load(random_state):
        """Create a random dataset for binary classification."""
        rs = int(random_state)
        x, y = make_classification(random_state=rs)
        return x, y
    
    
    @step(name="data_split")
    def split(x, y):
        """Split the data into train and test sets."""
        x, x_test, y, y_test = train_test_split(x, y, test_size=0.1)
        return x, x_test, y, y_test
    
    
    @step(name="model_training")
    def train(x, x_test, y, training_iterations):
        """Train a Logistic Regression model."""
        iters = int(training_iterations)
        model = LogisticRegression(max_iter=iters)
        model.fit(x, y)
        print(model.predict(x_test))
    
    
    @pipeline(name="binary-classification", experiment="kale-tutorial")
    def ml_pipeline(rs=42, iters=100):
        """Run the ML pipeline."""
        x, y = load(rs)
        x, x_test, y, y_test = split(x, y)
        train(x, x_test, y, iters)
    
    
    if __name__ == "__main__":
        ml_pipeline(rs=42, iters=100)
    

    Alternatively, download the kale_sdk.py Python file.

    In this code sample, you start with a standard Python script that trains a Logistic Regression model. Moreover, you have decorated the functions using the Kale SDK. To read more about how to create this file, head to the corresponding Kale SDK user guide.

  5. Create a new function that will be a step in the pipeline and it will create the HTML artifact. To achieve this, decorate the function with the step and artifact decorators. The artifact decorator takes two arguments; the name of the step and the absolute path where the HTML artifact is stored. The following snippet summarizes the changes in the code:

    --- examples/sdk/sdk.py
    +++ examples/artifacts/decorator.py
    @@ -5,7 +5,7 @@
     This script trains an ML pipeline to solve a binary classification task.
     """
     
    -from kale.sdk import pipeline, step
    +from kale.sdk import artifact, pipeline, step
     from sklearn.datasets import make_classification
     from sklearn.linear_model import LogisticRegression
     from sklearn.model_selection import train_test_split
    @@ -24,6 +24,13 @@
         """Split the data into train and test sets."""
         x, x_test, y, y_test = train_test_split(x, y, test_size=0.1)
         return x, x_test, y, y_test
    +
    +
    +@artifact(name="plot", path="/home/jovyan/plot.html")
    +@step(name="plot_data")
    +def plot(x, y):
    +    """Create an HTML artifact for KFP UI."""
    +    pass
     
     
     @step(name="model_training")
    

    Copy the resulting code below or download the kale_artifacts_decorator.py Python file.

  6. Create an HTML figure inside the function and save it as plot.html. Then, call the plot function inside the ml_pipelines function. The following snippet summarizes the changes in the code:

    --- examples/artifacts/decorator.py
    +++ examples/artifacts/artifacts.py
    @@ -5,6 +5,10 @@
     This script trains an ML pipeline to solve a binary classification task.
     """
     
    +import base64
    +from io import BytesIO
    +
    +import matplotlib.pyplot as plt
     from kale.sdk import artifact, pipeline, step
     from sklearn.datasets import make_classification
     from sklearn.linear_model import LogisticRegression
    @@ -30,7 +34,17 @@
     @step(name="plot_data")
     def plot(x, y):
         """Create an HTML artifact for KFP UI."""
    -    pass
    +    fig = plt.figure()
    +    ax = fig.add_subplot(1, 1, 1)
    +    ax.scatter(x[:, 0], y)
    +
    +    tmpfile = BytesIO()
    +    fig.savefig(tmpfile, format='png')
    +    encoded = base64.b64encode(tmpfile.getvalue()).decode('utf-8')
    +
    +    html = '<img src=\'data:image/png;base64,{}\'>'.format(encoded)
    +    with open('plot.html', 'w') as f:
    +        f.write(html)
     
     
     @step(name="model_training")
    @@ -47,6 +61,7 @@
         """Run the ML pipeline."""
         x, y = load(rs)
         x, x_test, y, y_test = split(x, y)
    +    plot(x, y)
         train(x, x_test, y, iters)
     
     
    

    Copy the resulting code below or download the kale_artifacts.py Python file.

    Warning

    If the path does not point to a valid file, the step will fail with an error.

    Note

    You can generate more than one artifact per step by applying the same decorator multiple times:

    @artifact(name="plot_1", path="./plot_1.html")
    @artifact(name="plot_2", path="./plot_2.html")
    @step(name="plot_data")
    def plot(x, y):
        ...
    

    Note

    This example assumes that you are running the Python script from your /home directory, in a Notebook server. If you change this you should also update the path argument of the artifact decorator accordingly.

  7. Run the script locally to test whether your code runs successfully using Kale’s marshalling mechanism:

    $ python3 -m kale kale_artifacts.py
    
  8. (Optional) Produce a workflow YAML file that you can inspect:

    $ python3 -m kale kale_artifacts.py --compile
    

    After the successful execution of this command, look for the workflow YAML file inside a .kale directory inside your working directory. This is a file that you could upload and submit to Kubeflow manually through its User Interface (KFP UI).

  9. Deploy and run your code as a KFP pipeline:

    $ python3 -m kale kale_artifacts.py --kfp
    

    Note

    To see the complete list of arguments and their respective usage run python3 -m kale --help.

  10. Navigate to the KFP UI and observe the HTML Artifact you created inside the Visualizations tab of the plot_data step:

    ../../../_images/artifacts.png

Summary

You have successfully created a KFP HTML artifact depicting a simple matplotlib figure to visualize through the KFP UI.

What’s Next

Check out the rest of the Kale user guides.