Configure KServe

In this section you will configure KServe for exposing your served models to the outer world.

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:/# cd ~/ops/deployments
  2. Specify the type of serving. Choose one of the following options based on whether you want host-based or path-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_TYPE=path-based
    root@rok-tools:~/ops/deployments# export SERVING_TYPE=host-based
  3. Specify the serving subdomain:

    root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=<SUBDOMAIN>

    Replace <SUBDOMAIN> with your serving subdomain, for example:

    root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=arrikto-cluster-serving.serving.example.com
  4. Specify the serving prefix. Choose one of the following options based on whether you use path-based or host-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_PREFIX=<PREFIX>

    Replace <PREFIX> with your desired prefix, for example:

    root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/serving
    root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/
  5. Set the KServe URL template. Choose one of the following options based on whether you use path-based or host-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"
    root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE=
  6. Render the patch for KServe:

    root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml.j2 \ > -o kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml
  7. Edit kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/kustomization.yaml and ensure that the ingress patch is enabled:

    patches: - path: patches/ingress.yaml
  8. Render the patch for Knative:

    root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml.j2 \ > -o kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml
  9. Edit kubeflow/manifests/common/knative/knative-serving/overlays/deploy/kustomization.yaml and ensure that the config-domain patch is enabled:

    patches: - path: patches/config-domain.yaml
  10. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit -am "Configure KServe"
  11. Apply the manifests:

    root@rok-tools:~/ops/deployments# rok-deploy --apply \ > kubeflow/manifests/common/knative/knative-serving/overlays/deploy \ > kubeflow/manifests/contrib/kserve/kserve/overlays/deploy

Verify

  1. Ensure that the urlTemplate for KServe is equal to the one you previously set. Choose one of the following options based on whether you use path-based or host-based serving.

    root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate "https://arrikto-cluster-serving.serving.example.com/serving/{{ .Namespace }}/{{ .Name }}"
    root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate ""
  2. Ensure that you have configured Knative for you serving subdomain. Ensure that the output of the following command is equal to your serving subdomain:

    root@rok-tools:~# kubectl get configmap config-domain \ > -n knative-serving -o jsonpath={.data} \ > | jq keys[] "arrikto-cluster-serving.serving.example.com"

Summary

You have successfully configured KServe for exposing your served models to the outside world.

What’s Next

The next step is to expose Istio, our service mesh.