Upgrade Kernel Modules

In EKF 1.5.2, by upgrading Rok, you have also upgraded the kernel module that Rok uses to track the blocks that have been modified.

The former kernel module Rok used for changed block tracking had a bug and, under certain circumstances, didn’t properly commit its metadata when Rok snapshotted the device. This could have lead to the corruption of the CBT metadata, which could result in Rok snapshots failing.

Because the old version of the kernel module might be in use by existing devices, upgrading it requires rebooting all the nodes in your cluster.

Although snapshots that have completed successfully should not be affected, to eliminate any chances of corruption, you should also reset the CBT metadata of all Rok volumes.

This guide will walk you through rebooting all cluster nodes and resetting the CBT metadata of all Rok volumes, so that the new version of the kernel module can take effect.

Fast Forward

This feature was introduced in EKF 1.5.2. You should only run this guide if you are upgrading from an EKF version older than 1.5.2 to version 1.5.2 or later. If you are upgrading EKF from version 1.5.2 or later, proceed to the What’s Next section.

Procedure

  1. Go to your rok-tools management environment.

  2. Reboot all cluster nodes. Choose one of the following options based on the service in which you have created your management environment:

    Kubernetes

    Docker

    Cloud Instance

    1. Find the node where rok-tools runs. You have to restart this node last:

      root@rok-tools-0:~# export ROK_TOOLS_NODE=$(kubectl get pods -A -l app=rok-tools --no-headers -o custom-columns=NODE:.spec.nodeName)

    2. Run rok-k8s-reboot and follow the on-screen instructions to reboot all cluster nodes one by one, except the one where rok-tools runs:

      root@rok-tools-0:~# rok-k8s-reboot \ > --nodes ALL \ > --exclude-nodes ${ROK_TOOLS_NODE:?}

      Air Gapped

      Follow the on-screen instructions to specify your internal registry, where you have mirrored the images.

    3. Note the node where rok-tools runs. You will need it at a later step:

      root@rok-tools-0:~# echo ${ROK_TOOLS_NODE:?} ip-192-168-182-36.eu-central-1.compute.internal

    4. Run rok-k8s-reboot and follow the on-screen instructions to reboot the node where rok-tools runs:

      root@rok-tools-0:~# rok-k8s-reboot \ > --nodes ${ROK_TOOLS_NODE:?} \ > --exclude-nodes None

      Air Gapped

      Follow the on-screen instructions to specify your internal registry, where you have mirrored the images.

    5. When the node reboots you will lose your connection to the rok-tools Pod. Wait until rok-tools is up and running again:

      # watch kubectl get pods -A -l app=rok-tools Every 2.0s: kubectl get pods -A -l app=rok-tools rok-tools: Fri Jun 10 09:49:08 2022 NAMESPACE NAME READY STATUS RESTARTS AGE default rok-tools-0 1/1 Running 1 12m

    6. Go to your rok-tools management environment

    7. Run rok-k8s-reboot and follow the on-screen instructions to finalize rebooting the node where rok-tools used to run. Replace <ROK_TOOLS_NODE> with the node name you noted at step 2c:

      root@rok-tools-0:~# rok-k8s-reboot \ > --nodes <ROK_TOOLS_NODE> \ > --exclude-nodes None

      Air Gapped

      Follow the on-screen instructions to specify your internal registry, where you have mirrored the images.

    1. Run rok-k8s-reboot and follow the on-screen instructions to reboot all cluster nodes one by one:

      root@rok-tools:~# rok-k8s-reboot \ > --nodes ALL \ > --exclude-nodes None

      Air Gapped

      Follow the on-screen instructions to specify your internal registry, where you have mirrored the images.

    1. Run rok-k8s-reboot and follow the on-screen instructions to reboot all cluster nodes one by one:

      root@rok-tools:~# rok-k8s-reboot \ > --nodes ALL \ > --exclude-nodes None

      Air Gapped

      Follow the on-screen instructions to specify your internal registry, where you have mirrored the images.

  3. Wait until Rok kmod has become ready. Verify field READY is 1/1 and field STATUS is Running for all Pods:

    root@rok-tools:~# watch kubectl get pods -n rok-system -l app=rok-kmod Every 2.0s: kubectl get pods -n rok-system -l app=rok-kmod rok-tools: Wed Jun 8 09:39:58 2022 NAME READY STATUS RESTARTS AGE rok-kmod-j9bpw 1/1 Running 0 1m rok-kmod-pqbxb 1/1 Running 0 1m

  4. Run rok-reset-cbt and follow the on-screen instructions to reset the CBT metadata of all Rok PVCs in all namespaces:

    root@rok-tools:~# rok-reset-cbt \ > --pvcs ALL \ > --namespace ALL \ > --storage-class rok

Summary

You have successfully upgraded the kernel modules used by Rok and reset the CBT metadata of all Rok volumes.

What’s Next

The next step is to deploy Rok Scheduler.