Configure KServe

In this section you will configure KServe for exposing your served models to the outer world.

Fast Forward

If you have already configured KServe, expand this box to fast-forward.

1. Specify the type of serving you want. Choose one of the following options based on whether you use path-based or host-based serving.

   Path-based

   Host-based

   

   root@rok-tools:~# export SERVING_TYPE=path-based

   

   root@rok-tools:~# export SERVING_TYPE=host-based

2. Set the serving subdomain:

   

   root@rok-tools:~# export SERVING_SUBDOMAIN=${SERVING_EKS_ALB_ACM_SUBDOMAIN?}

3. Specify the serving prefix. Choose one of the following options based on whether you use path-based or host-based serving.

   Path-based

   Host-based

   

   root@rok-tools:~# export SERVING_PREFIX=<PREFIX>

   Replace <PREFIX> with your desired prefix, for example:

   

   root@rok-tools:~# export SERVING_PREFIX=/serving

   

   root@rok-tools:~# export SERVING_PREFIX=/

4. Set the KServe URL template. Choose one of the following options based on whether you use path-based or host-based serving.

   Path-based

   Host-based

   

   root@rok-tools:~# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"

   

   root@rok-tools:~# export SERVING_URL_TEMPLATE=

5. Go to your GitOps repository, inside your rok-tools management environment:

   

   root@rok-tools:/# cd ~/ops/deployments

6. Save your state:

   

   root@rok-tools:~/ops/deployments# j2 deploy/env.serving.j2 \ > -o deploy/env.serving

7. Proceed to the Verify section.

Choose one of the following options:

• Option 1: Configure KServe Automatically (preferred).
• Option 2: Configure KServe Manually.

What You’ll Need

• A configured management environment.
• An existing EKS cluster.

Option 1: Configure KServe Automatically (preferred)

Under construction

This section is a work in progress. Use Option 2: Configure KServe Manually instead.

Configure KServe by following the on-screen instructions on the rok-deploy user interface.

If rok-deploy is not already running, start it with:

root@rok-tools:~# rok-deploy --run-from serving-eks-alb-kserve

Proceed to the Summary section.

Option 2: Configure KServe Manually

If you want to configure KServe manually, follow the instructions below.

Procedure

1. Go to your GitOps repository, inside your rok-tools management environment:

   

   root@rok-tools:/# cd ~/ops/deployments

2. Restore the required context from previous sections:

   

   root@rok-tools:~/ops/deployments# source deploy/env.serving-eks-alb-acm
   

   root@rok-tools:~/ops/deployments# export SERVING_EKS_ALB_ACM_SUBDOMAIN

3. Specify the type of serving you want. Choose one of the following options based on whether you want path-based or host-based serving.

   Path-based

   Host-based

   

   root@rok-tools:~/ops/deployments# export SERVING_TYPE=path-based

   

   root@rok-tools:~/ops/deployments# export SERVING_TYPE=host-based

4. Set the serving subdomain:

   

   root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=${SERVING_EKS_ALB_ACM_SUBDOMAIN?}

   Note

   Use the one you have specified when creating the SSL certificate.

5. Specify the serving prefix. Choose one of the following options based on whether you use path-based or host-based serving.

   Path-based

   Host-based

   

   root@rok-tools:~/ops/deployments# export SERVING_PREFIX=<PREFIX>

   Replace <PREFIX> with your desired prefix, for example:

   

   root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/serving

   

   root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/

6. Set the KServe URL template. Choose one of the following options based on whether you use path-based or host-based serving.

   Path-based

   Host-based

   

   root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"

   

   root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE=

7. Render the patch for KServe:

   

   root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml.j2 \ > -o kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml

8. Edit kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/kustomization.yaml and ensure that the ingress patch is enabled:

   

   patches: - path: patches/ingress.yaml

9. Render the patch for Knative:

   

   root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml.j2 \ > -o kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml

10. Edit kubeflow/manifests/common/knative/knative-serving/overlays/deploy/kustomization.yaml and ensure that the config-domain patch is enabled:

    

    patches: - path: patches/config-domain.yaml

11. Save your state:

    

    root@rok-tools:~/ops/deployments# j2 deploy/env.serving.j2 \ > -o deploy/env.serving

12. Commit your changes:

    

    root@rok-tools:~/ops/deployments# git commit -am "Configure KServe"

13. Apply the manifests:

    

    root@rok-tools:~/ops/deployments# rok-deploy --apply \ > kubeflow/manifests/common/knative/knative-serving/overlays/deploy \ > kubeflow/manifests/contrib/kserve/kserve/overlays/deploy

Verify

1. Ensure that the urlTemplate for KServe is equal to the one you previously set. Choose one of the following options based on whether you use path-based or host-based serving.

   Path-based

   Host-based

   

   root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate "https://arrikto-cluster-serving.serving.example.com/serving/{{ .Namespace }}/{{ .Name }}"

   

   root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate ""

2. Ensure that you have configured Knative for you serving subdomain. Ensure that the output of the following command is equal to your serving subdomain:

   

   root@rok-tools:~# kubectl get configmap config-domain \ > -n knative-serving -o jsonpath={.data} \ > | jq keys[] "arrikto-cluster-serving.serving.example.com"

Summary

You have successfully configured KServe for exposing your served models to the outside world.

What’s Next

The next step is to expose Istio.

• Expose Istio