Protect Rok System Pods

This guide describes the necessary steps to patch an existing Rok cluster on Kubernetes, in order to protect essential Rok System Pods from being terminated in case of a memory pressure scenario.

Procedure

  1. Get the version of your Rok Operator:

    root@rok-tools:~# kubectl get -n rok-system sts rok-operator --no-headers \ > -o custom-columns=:.spec.template.spec.containers[0].image gcr.io/arrikto-deploy/rok-operator:release-1.1-l0-release-1.1

    If the image tag of your Rok Operator is release-1.1-l0-release-1.1 or newer, you may proceed to the Verify section.

  2. Watch the rok-csi-controller logs and ensure that no pipelines or snapshot policies are running, namely nothing will be logged for 30 secs:

    root@rok-tools:~# kubectl -n rok logs -l app=rok-csi-controller -c csi-controller -f --tail=100

  3. Scale down the rok-operator StatefulSet:

    root@rok-tools:~# kubectl -n rok-system scale sts rok-operator --replicas=0 statefulset.apps/rok-operator scaled

  4. Ensure rok-operator has scaled down to zero:

    root@rok-tools:~# kubectl -n rok-system get sts rok-operator NAME READY AGE rok-operator 0/0 2h

  5. Scale down the rok-csi-controller StatefulSet:

    root@rok-tools:~# kubectl -n rok scale sts rok-csi-controller --replicas=0 statefulset.apps/rok-csi-controller scaled

  6. Ensure rok-csi-controller has scaled down to zero:

    root@rok-tools:~# kubectl get -n rok sts rok-csi-controller NAME READY AGE rok-csi-controller 0/0 2h

  7. Watch the rok-csi-node logs and ensure that all pending operations have finished, namely nothing will be logged for 30 secs:

    root@rok-tools:~# kubectl -n rok logs -l app=rok-csi-node -c csi-node -f --tail=100

  8. Delete the rok-csi-node DaemonSet:

    root@rok-tools:~# kubectl -n rok delete ds rok-csi-node daemonset.apps "rok-csi-node" deleted

  9. Specify the image for the new rok-operator, which will assign the system-node-critical Priority Class to all Rok and Rok CSI resources:

    root@rok-tools:~# export ROK_OPERATOR_IMAGE=gcr.io/arrikto-deploy/rok-operator:release-1.1-l0-release-1.1

  10. Patch rok-operator to pull the new image:

    root@rok-tools:~# kubectl -n rok-system patch sts rok-operator \ > --patch "{\"spec\": {\"template\": {\"spec\": {\"containers\": [{\"name\": \"rok-operator\", \"image\": \"${ROK_OPERATOR_IMAGE}\"}]}}}}" statefulset.apps/rok-operator patched

  11. Scale back up rok-operator to its initial size to recreate the Rok and Rok CSI resources:

    root@rok-tools:~# kubectl -n rok-system scale sts rok-operator --replicas=1 statefulset.apps/rok-operator scaled

Verify

  1. Ensure that the Rok cluster is up and running:

    root@rok-tools:~# watch kubectl get rokcluster -n rok NAME VERSION HEALTH TOTAL MEMBERS READY MEMBERS PHASE AGE rok release-1.1-l0-release-1.1-rc5 OK 3 3 Running 2h

  2. Ensure that rok, rok-csi-node, rok-csi-guard now have the system-node-critical Priority Class:

    root@rok-tools:~# kubectl get -n rok daemonset rok --no-headers \ > -o custom-columns=:.spec.template.spec.priorityClassName system-node-critical
    root@rok-tools:~# kubectl get -n rok sts rok-csi-node --no-headers \ > -o custom-columns=:.spec.template.spec.priorityClassName system-node-critical
    root@rok-tools:~# kubectl get -n rok deploy rok-csi-guard --no-headers \ > -o custom-columns=:.spec.template.spec.priorityClassName system-node-critical

  3. Ensure that rok-csi-controller now has the system-cluster-critical Priority Class:

    root@rok-tools:~# kubectl get -n rok sts rok-csi-controller --no-headers \ > -o custom-columns=:.spec.template.spec.priorityClassName system-cluster-critical

Summary

You have successfully patched all Rok System Pods with the highest pre-defined Kubernetes Priority Classes and have protected them against evictions and terminations under memory pressure scenarios.

What’s Next

The next step is to protect the Rok External Services Pods.