Upgrade Istio¶
EKF 2.0.X uses Istio 1.14.3. This guide will walk you through upgrading Istio to version 1.14.3 from Istio 1.9.6 that the previous version of EKF was using.
Fast Forward
If you are upgrading from EKF 2.0 or later, expand this box to fast-forward.
- Proceed to the Verify section.
What You’ll Need¶
- An upgraded management environment.
- Your clone of the Arrikto GitOps repository.
- Arrikto manifests for EKF version 2.0.2.
- An upgraded Rok and Kubeflow deployment.
Procedure¶
Go to your GitOps repository inside your
rok-tools
management environment:root@rok-tools:~# cd ~/ops/deploymentsDelete the previous Istio control plane installation:
root@rok-tools:~/ops/deployments# rok-deploy --delete \ > rok/rok-external-services/istio/istio-1-9/istio-install/overlays/deploy \ > rok/rok-external-services/istio/istio-1-9/knative-serving/overlays/deployApply the new Istio control plane:
root@rok-tools:~/ops/deployments# rok-deploy --apply \ > rok/rok-external-services/istio/istio-1-14/istio-crds/overlays/deploy \ > rok/rok-external-services/istio/istio-1-14/istio-namespace/overlays/deploy \ > rok/rok-external-services/istio/istio-1-14/istio-install/overlays/deploy \ > rok/rok-external-services/istio/istio-1-14/knative-serving/overlays/deployRemove the deprecated Istio-related resources left by the previous version of EKF:
root@rok-tools:~/ops/deployments# rok-kf-prune --app istioConfirm that the
knative-serving
andkubeflow
namespaces, as well as all of the kubeflow user namespaces (namespaces that start withkubeflow-
) have Istio sidecar injection enabled. Ensure that these namespaces show up in the following command’s output:root@rok-tools:~/ops/deployments# kubectl get ns -l istio-injection=enabled NAME STATUS AGE knative-serving Active 5d16h kubeflow Active 5d16h kubeflow-user Active 5d16h ...Upgrade the Istio sidecars, by deleting all Pods in the namespaces you found above. Istio will inject the new version sidecar once the owning controllers recreate the deleted Pods:
root@rok-tools:~/ops/deployments# kubectl get ns -l istio-injection=enabled --no-headers \ > | awk '{print $1}' \ > | xargs -n1 -I {} kubectl delete pod --all -n {}Copy the patches from the old kustomization to the new one. Specifically:
Copy the
ingress-host
patch:root@rok-tools:~/ops/deployments# cp -av \ > rok/rok-external-services/istio/istio-1-9/istio-install/overlays/deploy/patches/ingress-host.yaml \ > rok/expose-ekf/overlays/deploy/patches/ingress-host.yamlCopy the
ingress-tls
patch:root@rok-tools:~/ops/deployments# cp -av \ > rok/rok-external-services/istio/istio-1-9/istio-install/overlays/deploy/patches/ingress-tls.yaml \ > rok/expose-ekf/overlays/deploy/patches/ingress-tls.yamlUpdate the copied
ingress-tls
patch to use the newapiVersion
for Ingress resources:root@rok-tools:~/ops/deployments# sed -i \ > -e 's|extensions/v1beta1|networking.k8s.io/v1|' \ > rok/expose-ekf/overlays/deploy/patches/ingress-tls.yamlCopy the
certificate
patch:root@rok-tools:~/ops/deployments# cp -av \ > rok/rok-external-services/istio/istio-1-9/istio-install/overlays/deploy/patches/certificate.yaml \ > rok/expose-ekf/overlays/deploy/patches/certificate.yaml
Copy the secrets from the old kustomization to the new one:
root@rok-tools:~/ops/deployments# cp -av \ > rok/rok-external-services/istio/istio-1-9/istio-install/overlays/deploy/secrets/ \ > rok/expose-ekf/overlays/deploy/Edit the
rok/expose-ekf/overlays/deploy/kustomization.yaml
file and enable the necessary resources and patches, as you see them in the snippets below. Choose one of the following options based on your cloud provider, the type of the load balancer you are using, and who manages your SSL certificates:resources: - ../../base - ../../base/ingress.yaml patches: - path: patches/ingress-host.yaml target: kind: Ingress name: istio-ingress - path: patches/xff.yaml target: kind: EnvoyFilter name: xff-trust-hopsresources: - ../../base - ../../base/ingress.yaml - ../../base/certificate.yaml patches: - path: patches/ingress-host.yaml target: kind: Ingress name: istio-ingress - path: patches/ingress-tls.yaml - path: patches/certificate.yaml - path: patches/xff.yaml target: kind: EnvoyFilter name: xff-trust-hopsresources: - ../../base - ../../base/ingress.yaml patches: - path: patches/ingress-host.yaml target: kind: Ingress name: istio-ingress - path: patches/ingress-tls.yaml - path: patches/xff.yaml target: kind: EnvoyFilter name: xff-trust-hops secretGenerator: - name: istio-ingress-tls-secret files: - secrets/tls.crt - secrets/tls.key type: "kubernetes.io/tls"resources: - ../../base - ../../base/ingress.yaml - ../../base/certificate.yaml patches: - path: patches/ingress-host.yaml target: kind: Ingress name: istio-ingress - path: patches/ingress-tls.yaml - path: patches/certificate.yaml - path: patches/xff.yaml target: kind: EnvoyFilter name: xff-trust-hopsresources: - ../../base - ../../base/ingress.yaml - ../../base/certificate.yaml patches: - path: patches/ingress-host.yaml target: kind: Ingress name: istio-ingress - path: patches/ingress-tls.yaml - path: patches/certificate.yaml - path: patches/xff.yaml target: kind: EnvoyFilter name: xff-trust-hopsresources: - ../../base - ../../base/ingress.yaml patches: - path: patches/ingress-host.yaml target: kind: Ingress name: istio-ingress - path: patches/ingress-tls.yaml - path: patches/xff.yaml target: kind: EnvoyFilter name: xff-trust-hops secretGenerator: - name: istio-ingress-tls-secret files: - secrets/tls.crt - secrets/tls.key type: "kubernetes.io/tls"Edit the
rok/expose-ekf/overlays/deploy/patches/xff.yaml
file.Set the number of trusted proxies in front of the Istio Gateway. Choose one of the following options based on your cloud provider and the type of load balancer your are using:
root@rok-tools:~/ops/deployments# export TRUSTED_FRONT_PROXIES=2root@rok-tools:~/ops/deployments# export TRUSTED_FRONT_PROXIES=1root@rok-tools:~/ops/deployments# export TRUSTED_FRONT_PROXIES=1root@rok-tools:~/ops/deployments# export TRUSTED_FRONT_PROXIES=1Render the Istio envoy filter patch template with the values you have specified:
root@rok-tools:~/ops/deployments# j2 \ > rok/expose-ekf/overlays/deploy/patches/xff.yaml.j2 \ > -o rok/expose-ekf/overlays/deploy/patches/xff.yaml
Commit your changes:
root@rok-tools:~/ops/deployments# git commit -am "Expose Istio"Apply the kustomization:
root@rok-tools:~/ops/deployments# rok-deploy --apply rok/expose-ekf/overlays/deployAir Gapped
In case of a private only EKS cluster, external-dns will not be able to update Route 53 entries automatically. So you have to manually create an alias record, i.e., an A record pointing to the internal ALB created by the AWS Load Balancer Controller.
Verify¶
Verify that the Pods in the
istio-system
namespace are up and running. Check the Pod status, and verify field STATUS is Running and field READY is 1/1 for all Pods:root@rok-tools:~# kubectl -n istio-system get pods NAME READY STATUS RESTARTS AGE authservice-0 1/1 Running 0 1m istio-ingressgateway-57f58bf544-x45kw 1/1 Running 0 1m istiod-68f6c899f5-wzjfm 1/1 Running 0 1mVerify that you have successfully created the Ingress object for Istio. The HOSTS field should match your SUBDOMAIN and the ADDRESS field shows the hostname of your Load Balancer:
root@rok-tools:~/ops/deployments# kubectl -n istio-system get ingress NAME CLASS HOSTS ADDRESS PORTS AGE istio-ingress nginx arrikto-cluster.apps.example.com e53a524a-ingressnginx-ingr-8872-592794601.us-east-1.elb.amazonaws.com 80 1mOpen your browser, and go to the EKF UI at
https://<YOUR_SUBDOMAIN>/Replace
<YOUR_SUBDOMAIN>
with your the value shown in HOSTS above. For example:https://arrikto-cluster.apps.example.com/Air Gapped
Use dynamic port forwarding along with SOCKS5 protocol in your browser.
Summary¶
You have successfully upgraded Istio.