HP Tuning with the Kale SDK

This section will guide you through configuring and running a Katib experiment using the Kale SDK, to tune the hyperparameters (HP) of your Machine Learning (ML) model.

What You’ll Need

  • An EKF or MiniKF deployment with the default Kale Docker image.
  • An understanding of how the Kale SDK works.

Procedure

  1. Create a new Notebook server using the default Kale Docker image. The image will have the following naming scheme:

    gcr.io/arrikto/jupyter-kale-py36:<IMAGE_TAG>
    

    Note

    The <IMAGE_TAG> varies based on the MiniKF or EKF release.

  2. Connect to the server, open a terminal, and install scikit-learn:

    $ pip3 install --user scikit-learn==0.23.0
    
  3. Create a new python file and name it kale_katib.py:

    $ touch kale_katib.py
    
  4. Copy and paste the following code inside kale_katib.py:

    # Copyright © 2021 Arrikto Inc.  All Rights Reserved.
    
    """Kale SDK.
    
    This script trains an ML pipeline to solve a binary classification task.
    """
    
    from kale.sdk import has_metrics, pipeline, step
    from kale.sdk.logging import log_metric
    from sklearn.datasets import make_classification
    from sklearn.linear_model import LogisticRegression
    from sklearn.metrics import accuracy_score
    from sklearn.model_selection import train_test_split
    
    
    @step(name="data_loading")
    def load(random_state):
        """Create a random dataset for binary classification."""
        rs = int(random_state)
        x, y = make_classification(random_state=rs)
        return x, y
    
    
    @step(name="data_split")
    def split(x, y):
        """Split the data into train and test sets."""
        x, x_test, y, y_test = train_test_split(x, y, test_size=0.1)
        return x, x_test, y, y_test
    
    
    @step(name="model_training")
    def train(x, x_test, y, training_iterations):
        """Train a Logistic Regression model."""
        iters = int(training_iterations)
        model = LogisticRegression(max_iter=iters)
        model.fit(x, y)
        return model
    
    
    @has_metrics
    @step(name="model_evaluation")
    def evaluate(model, x_test, y_test):
        """Evaluate the model on the test dataset."""
        y_pred = model.predict(x_test)
        accuracy = accuracy_score(y_test, y_pred)
        log_metric(name="accuracy", value=accuracy)
    
    
    @pipeline(name="binary-classification", experiment="kale-tutorial")
    def ml_pipeline(rs=42, iters=100):
        """Run the ML pipeline."""
        x, y = load(rs)
        x, x_test, y, y_test = split(x, y)
        model = train(x, x_test, y, iters)
        evaluate(model, x_test, y_test)
    
    
    if __name__ == "__main__":
        ml_pipeline(rs=42, iters=100)
    

    Alternatively, download the kale_katib_starter_code.py Python file.

    In this code sample, you start with a standard Python script that trains a Logistic Regression model. Moreover, you have decorated the functions using the Kale SDK. To read more about how to create this file, head to the corresponding KFP metrics user guide.

  5. Define the Katib experiment configuration using the Katib SDK. The following snippet summarizes the changes in code:

    --- examples/metrics.py
    +++ examples/configuration.py
    @@ -11,6 +11,35 @@
     from sklearn.linear_model import LogisticRegression
     from sklearn.metrics import accuracy_score
     from sklearn.model_selection import train_test_split
    +from kubeflow import katib
    +
    +
    +# The Katib experiment definition.
    +katib_experiment = katib.V1beta1ExperimentSpec(
    +    max_trial_count=3,
    +    parallel_trial_count=1,
    +    max_failed_trial_count=0,
    +    algorithm=katib.V1beta1AlgorithmSpec(
    +        algorithm_name="grid"
    +    ),
    +    objective=katib.V1beta1ObjectiveSpec(
    +        type="maximize",
    +        objective_metric_name="accuracy"
    +    ),
    +    parameters=[katib.V1beta1ParameterSpec(
    +        name="c",
    +        parameter_type="double",
    +        feasible_space=katib.V1beta1FeasibleSpace(
    +            min="0.1",
    +            max="1.0",
    +            step="0.3"),
    +    ), katib.V1beta1ParameterSpec(
    +        name="penalty",
    +        parameter_type="categorical",
    +        feasible_space=katib.V1beta1FeasibleSpace(
    +            list=["l2", "none"])
    +    )],
    +)
     
     
     @step(name="data_loading")
    

    Copy the resulting code below or download the kale_katib_conf.py Python file.

    A Katib experiment configuration has four main sections:

    • Trials: The maximum trials that Katib will start, how many of them will run in parallel, and after how many failed Trials the experiment will terminate.
    • Algorithm: The algorithm that you want to use for HP tuning (e.g., grid), specified as a katib.V1beta1AlgorithmSpec spec object.
    • Metric: The objective name that you use as an end goal (e.g., accuracy) specified as a katib.V1beta1ObjectiveSpec spec object. Note that the name of the objective should match one of the metrics you log with the log_metrics API. See the guide on how to produce KFP metrics for more details.
    • Hyperparameters: The names of the HPs you want to optimize as a list of katib.V1beta1ParameterSpec spec objects. For each HP, you should specify the feasible space inside a katib.V1beta1FeasibleSpace spec object. This can take the form of a numerical range (e.g., see the HP named c in the above code snippet) or a list of possible values (e.g., see the HP named penalty in the above code snippet).

    Note

    The Jupyter Kale images in the EKF or MiniKF deployments already come with the Katib SDK installed.

  6. Pass the Katib configuration as an argument to the pipeline decorated function:

    --- examples/configuration.py
    +++ examples/experiment.py
    @@ -75,7 +75,8 @@
         log_metric(name="accuracy", value=accuracy)
     
     
    -@pipeline(name="binary-classification", experiment="kale-tutorial")
    +@pipeline(name="binary-classification", experiment="kale-tutorial",
    +          katib_experiment=katib_experiment)
     def ml_pipeline(rs=42, iters=100):
         """Run the ML pipeline."""
         x, y = load(rs)
    

    Copy the resulting code below or download the kale_katib_experiment.py Python file.

  7. Create a parameterized pipeline, which receives the HPs that you want to tune as inputs. To read more about creating parameterized pipelines with the Kale SDK head to relevant Kale SDK guide:

    --- examples/experiment.py
    +++ examples/katib.py
    @@ -58,10 +58,11 @@
     
     
     @step(name="model_training")
    -def train(x, x_test, y, training_iterations):
    +def train(x, x_test, y, training_iterations, c, penalty):
         """Train a Logistic Regression model."""
         iters = int(training_iterations)
    -    model = LogisticRegression(max_iter=iters)
    +    c = float(c)
    +    model = LogisticRegression(max_iter=iters, C=c, penalty=penalty)
         model.fit(x, y)
         return model
     
    @@ -77,13 +78,13 @@
     
     @pipeline(name="binary-classification", experiment="kale-tutorial",
               katib_experiment=katib_experiment)
    -def ml_pipeline(rs=42, iters=100):
    +def ml_pipeline(rs=42, c=1.0, penalty="l2", iters=100):
         """Run the ML pipeline."""
         x, y = load(rs)
         x, x_test, y, y_test = split(x, y)
    -    model = train(x, x_test, y, iters)
    +    model = train(x, x_test, y, iters, c, penalty)
         evaluate(model, x_test, y_test)
     
     
     if __name__ == "__main__":
    -    ml_pipeline(rs=42, iters=100)
    +    ml_pipeline(rs=42, c=1.0, penalty="l2", iters=100)
    

    Copy the resulting code below or download the kale_katib.py Python file.

  8. Run the script locally to test whether your code runs successfully using Kale’s marshalling mechanism.

    $ python3 -m kale kale_katib.py
    
  9. (Optional) Produce a workflow YAML file that you can inspect:

    $ python3 -m kale kale_katib.py --compile
    

    After the successful execution of this command, look for the workflow YAML file inside a .kale directory inside your working directory. This is a file that you could upload and submit to Kubeflow manually through its User Interface (KFP UI).

  10. Deploy and run your code as a KFP pipeline:

    $ python3 -m kale kale_katib.py --kfp
    

    Note

    To see the complete list of arguments and their respective usage, run python3 -m kale --help.

Summary

You have successfully created a Katib experiment using the Kale SDK.

What’s Next

Check out the rest of the Kale user guides.