Switch Between Host-based and Path-based Serving

This guide will walk you through switching between host-based and path-based serving.

  • Host-based: Each model is exposed under a different subdomain. This requires a wildcard certificate and a wildcard DNS entry. This option is the only one available in EKF 1.5.
  • Path-based: All models are exposed under a single subdomain, but under a different prefix. EKF 2.0 supports this option and we recommend you use it.

Important

This guide assumes that you have already exposed serving during the initial deployment. It also assumes that you will use the existing load balancer and TLS configuration.

What You’ll Need

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Specify the serving type you want. Choose one of the following options based on whether you want to use path-based or host-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_TYPE=path-based
    root@rok-tools:~/ops/deployments# export SERVING_TYPE=host-based
  3. Specify the serving subdomain:

    root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=<SUBDOMAIN>

    Replace <SUBDOMAIN> with your serving domain, for example:

    root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=arrikto-cluster-serving.serving.example.com

    Important

    This should be the same with the domain you used when generating your certificate.

  4. Specify the serving prefix. Choose one of the following options based on whether you want to use path-based or host-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_PREFIX=<PREFIX>

    Replace <PREFIX> with your desired prefix, for example:

    root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/serving
    root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/
  5. Set the KServe URL template. Choose one of the following options based on whether you want to use path-based or host-based serving.

    root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"
    root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE=
  6. Render the patch for KServe:

    root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml.j2 \ > -o kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml
  7. Edit kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/kustomization.yaml and ensure that the ingress patch is enabled:

    patches: - path: patches/ingress.yaml
  8. Render the patch for Knative:

    root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml.j2 \ > -o kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml
  9. Edit kubeflow/manifests/common/knative/knative-serving/overlays/deploy/kustomization.yaml and ensure that the config-domain patch is enabled:

    patches: - path: patches/config-domain.yaml
  10. Render the Istio ingress patch template with the variables you have specified:

    root@rok-tools:~/ops/deployments# j2 \ > rok/expose-serving/overlays/deploy/patches/ingress-host.yaml.j2 \ > -o rok/expose-serving/overlays/deploy/patches/ingress-host.yaml
  11. Edit rok/expose-serving/overlays/deploy/kustomization.yaml and ensure that the ingress-host patch is enabled:

    patches: - path: patches/ingress-host.yaml target: kind: Ingress name: knative-serving-ingress
  12. Render the patch for TLS termination. Choose one of the following options based on your cloud provider and the load balancer you are using.

    TLS termination takes place on ALB with an ACM certificate. You do not have to use a TLS patch, so skip this step.

    root@rok-tools:~/ops/deployments# j2 \ > rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml.j2 \ > -o rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml
    root@rok-tools:~/ops/deployments# j2 \ > rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml.j2 \ > -o rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml
    root@rok-tools:~/ops/deployments# j2 \ > rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml.j2 \ > -o rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml
  13. Optional

    If you are running on AWS EKS and have exposed serving with Application Load Balancer (ALB), save your new state:

    root@rok-tools:~/ops/deployments# j2 deploy/env.serving.j2 \ > -o deploy/env.serving
  14. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit -am "Switch Between Host-based and Path-based Serving"
  15. Apply the manifests:

    root@rok-tools:~/ops/deployments# rok-deploy --apply \ > kubeflow/manifests/common/knative/knative-serving/overlays/deploy \ > kubeflow/manifests/contrib/kserve/kserve/overlays/deploy \ > rok/expose-serving/overlays/deploy

Verify

  1. Ensure that the urlTemplate for KServe is equal to the one you previously set. Choose one of the following options based on whether you use path-based or host-based serving.

    root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate "https://arrikto-cluster-serving.serving.example.com/serving/{{ .Namespace }}/{{ .Name }}"
    root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate ""
  2. Ensure that you have configured Knative for you serving subdomain. Ensure that the output of the following command is equal to your serving subdomain:

    root@rok-tools:~# kubectl get configmap config-domain \ > -n knative-serving -o jsonpath={.data} \ > | jq keys[] "arrikto-cluster-serving.serving.example.com"
  3. Verify that you have successfully created the Ingress object for Istio. Ensure that the HOSTS field includes both your subdomain and your wildcard subdomain:

    root@rok-tools:~# kubectl get ingress -n knative-serving NAME CLASS HOSTS ADDRESS PORTS AGE knative-serving-ingress nginx-serving *.arrikto-cluster-serving.serving.example.com,arrikto-cluster-serving.serving.example.com e53a524a-ingressnginx-ingr-8872-592794601.us-east-1.elb.amazonaws.com 80 1m

Summary

You have successfully configured Serving to use either host-based or path-based serving.

What’s Next

Check out the rest of the operations you can perform on your Kubeflow deployment.