Rok v1.0 (unreleased)¶
This guide assumes that you have already cloned Arrikto’s deployment repository that contains needed Kubernetes manifests, i.e., https://github.com/arrikto/deployments.
Change your current directory to your local clone of Arrikto’s GitOps deployment repository. For example:
$ cd ~/ops/deployments
This guide uses the
deploy overlay in the commands to be executed since
this is the overlay that users are supposed to tailor based on their needs
Notable changes v0.15 -> v1.0¶
v0.15Rok integrates with Istio
v1.0Rok has been upgraded to work with Istio
- Rok tools now persists the whole of
/rootand not only
Upgrade your management environment¶
This version of Rok changes the way data is persisted in your management
environment: from now on,
rok-tools persists the whole of
not only the local GitOps repository but also user settings and credentials,
The location where the volume of
rok-tools is mounted has changed from
The steps below will instruct you how to mirror the local GitOps repo in a
private remote, so that you can later clone it into a new, empty
volume that the upgraded instance of
rok-tools will use to persist
First, add an extra remote of your choice so you can push local changes there:
root@rok-tools-0:/# cd ~/ops/deployments root@rok-tools-0:~/ops/deployments# git remote add private <repo-url>
Then, push your local GitOps repository to the remote you just added to safely keep your current changes. For example:
root@rok-tools-0:~/ops/deployments# git push private develop:develop
(Docker only) If you are running
rok-toolsas a local Docker container you need to clear the host directory that was previously used by Docker as the volume of
rok-tools. For example:
$ rm -rf rok-tools-data/*
Follow the steps described in the Upgrade your management environment section
Exec into the upgraded
$ kubectl exec -ti statefulset/rok-tools /bin/bash
$ docker exec -ti <ROK_TOOLS_CONTAINER_ID> /bin/bash
Since any changes under
/rootare lost, you have to reconfigure your management environment based on the individual subsections of the Configure your management environment section, as needed. From now on, changes in files under
/rootwill now be persisted in the external volume.
Restore your GitOps repository locally to make all your latest changes available in the new
root@rok-tools-0:/# mkdir ~/ops && cd ~/ops root@rok-tools-0:~/ops# git clone --branch develop --origin private <repo-url> deployments
Add the Arrikto provided remote:
root@rok-tools-0:/# cd ~/ops/deployments root@rok-tools-0:~/ops/deployments# git remote add origin email@example.com:arrikto/deployments.git
Update local repo:
root@rok-tools-0:~/ops/deployments# git fetch --all -p
Ensure your local branch tracks the Arrikto-managed one:
root@rok-tools-0:~/ops/deployments# git branch -u origin/develop
(Optional) Remove the private remote:
root@rok-tools-0:~/ops/deployments# git remote remove private
kubectlso that you can connect to your Kubernetes cluster. Depending on your environment and setup you might need to follow the Access EKS Cluster section or copy in your kubeconfig file.
(Kubernetes only) If you are running
rok-toolsas a Kubernetes StatefulSet you can optionally delete the old PVC, i.e.,
rok-ops-rok-tools-0, since the PVC has been renamed to
data-rok-tools-0and the old one is no longer needed.
Upgrade the local deployments repo as described in the Upgrade manifests section.
The upgrade procedure described below deviates from the generate/commit/apply model since requires some manual deletions.
v1.0 uses a newer version of Istio (
v1.5.7), containing many bug
fixes and improvements. In order to upgrade to version
1.5.7 from version
Delete the previous Istio control plane installation:
$ kubectl delete -k rok/rok-external-services/istio/istio-1-3-1/istio-install-1-3-1/overlays/deploy
Apply the new Istio control plane:
$ kubectl apply -k rok/rok-external-services/istio/istio-1-5-7/istio-crds-1-5-7/overlays/deploy $ kubectl apply -k rok/rok-external-services/istio/istio-1-5-7/istio-namespace-1-5-7/overlays/deploy $ kubectl apply -k rok/rok-external-services/istio/istio-1-5-7/istio-install-1-5-7/overlays/deploy
Check the Envoy Proxy sidecars that exist in the cluster:
$ istioctl proxy-status NAME CDS LDS EDS RDS PILOT VERSION activator-cfc66dc7-tzdzx.knative-serving SYNCED SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.3.1 autoscaler-6cc8bc459b-jghcg.knative-serving SYNCED SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.3.1 cluster-local-gateway-9d544d7db-c2xbq.istio-system STALE (Never Acknowledged) SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.3.1 istio-ingressgateway-74649669b7-x77tt.istio-system SYNCED SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.5.7 prometheus-6c846d79b9-r8v4b.istio-system SYNCED SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.5.7
The above list might include Kubeflow related components that will be upgraded later on.
This release of Rok comes with an upgraded version of Istio to fix an issue with the handling of
X-Forwarded-*headers by intermediate proxies, i.e., Rok URLs showed up as
httpURLs instead of
httpsones. If you have HTTP proxies in front of your Istio IngressGateway installation (e.g., ALB, NGINX, etc.), make sure to configure X-Forwarded-* settings.
Also have Ingress NGINX compute full X-Forwarded-For header by patching its configuration:
$ kubectl apply -f rok/nginx-ingress-controller/patch-configmap-l7.yaml
Upgrade Rok components¶
Upgrade Rok components using latest kustomizations as described in the Upgrade components section.
If you have integrated your deployment with Kubeflow then you have to upgrade Kubeflow to use new Istio.
To do so, first you have to manually delete some resources that cannot be updated. Specifically:
$ kubectl delete poddisruptionbudgets -n istio-system cluster-local-gateway $ kubectl delete deployment -n istio-system cluster-local-gateway
Also, delete KFServing Pods, so that Istio sidecars get upgraded:
$ kubectl delete pods -n knative-serving -l app=activator $ kubectl delete pods -n knative-serving -l app=autoscaler
Now upgrade Kubeflow as described in the Upgrade section.