Configure KServe¶
In this section you will configure KServe for exposing your served models to the outer world.
Fast Forward
If you have already configured KServe, expand this box to fast-forward.
- Proceed to the Verify section.
Overview
What You’ll Need¶
- A configured management environment.
- An existing EKS cluster.
- A working NGINX Ingress Controller deployment.
Procedure¶
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:/# cd ~/ops/deploymentsSpecify the type of serving you want. Choose one of the following options based on whether you want path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_TYPE=path-basedroot@rok-tools:~/ops/deployments# export SERVING_TYPE=host-basedSpecify the serving subdomain:
root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=${SERVING_AZURELB_DNS_NAME_LABEL?}.${AZURE_DEFAULTS_LOCATION?}.cloudapp.azure.comNote
This is the domain you used for your NGINX Ingress Controller.
Specify the serving prefix. Choose one of the following options based on whether you use path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_PREFIX=<PREFIX>Replace
<PREFIX>
with your desired prefix, for example:root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/servingroot@rok-tools:~/ops/deployments# export SERVING_PREFIX=/Set the KServe URL template. Choose one of the following options based on whether you use path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE=Render the patch for KServe:
root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml.j2 \ > -o kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yamlEdit
kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/kustomization.yaml
and ensure that theingress
patch is enabled:patches: - path: patches/ingress.yamlRender the patch for Knative:
root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml.j2 \ > -o kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yamlEdit
kubeflow/manifests/common/knative/knative-serving/overlays/deploy/kustomization.yaml
and ensure that theconfig-domain
patch is enabled:patches: - path: patches/config-domain.yamlCommit your changes:
root@rok-tools:~/ops/deployments# git commit -am "Configure KServe"Apply the manifests:
root@rok-tools:~/ops/deployments# rok-deploy --apply \ > kubeflow/manifests/common/knative/knative-serving/overlays/deploy \ > kubeflow/manifests/contrib/kserve/kserve/overlays/deploy
Verify¶
Ensure that the
urlTemplate
for KServe is the one you previously set. Choose one of the following options based on whether you use path-based or host-based serving.root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate "https://arrikto-cluster-serving.serving.example.com/serving/{{ .Namespace }}/{{ .Name }}"root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate ""Ensure that you have configured Knative for your serving subdomain. Ensure that the output of the following command is equal to your serving subdomain:
root@rok-tools:~# kubectl get configmap config-domain \ > -n knative-serving -o jsonpath={.data} \ > | jq keys[] "arrikto-cluster-serving.serving.example.com"