Configure KServe

In this section you will configure KServe for exposing your served models to the outside world.

What You’ll Need

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:/# cd ~/ops/deployments
  2. Specify the type of serving. Choose one of the following options based on whether you want path-based or host-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_TYPE=path-based

    Host-based serving is currently not supported.

  3. Set the subdomain to use for Serving:

    root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=${FQDN?}
  4. Specify the serving prefix. Choose one of the following options based on whether you use path-based or host-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_PREFIX=<PREFIX>

    Replace <PREFIX> with your desired prefix, for example:

    root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/serving

    Host-based serving is currently not supported.

  5. Set the KServe URL template. Choose one of the following options based on whether you use path-based or host-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"

    Host-based serving is currently not supported.

  6. Render the patch for KServe:

    root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml.j2 \ > -o kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml
  7. Edit kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/kustomization.yaml and ensure that the ingress patch is enabled:

    patches: - path: patches/ingress.yaml
  8. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit -am "Configure KServe"
  9. Apply the manifests:

    root@rok-tools:~/ops/deployments# rok-deploy --apply \ > kubeflow/manifests/contrib/kserve/kserve/overlays/deploy

Verify

  1. Ensure that the urlTemplate for KServe is equal to the one you previously set. Choose one of the following options based on whether you use path-based or host-based serving.

    root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate "https://serving.example.com/serving/{{ .Namespace }}/{{ .Name }}"

    Host-based serving is currently not supported.

Summary

You have successfully configured KServe for exposing your served models to the outside world.

What’s Next

The next step is to expose Istio, our service mesh.