Increase Rok Snapshot Space

In order to take snapshots of PersistentVolumeClaims, Rok needs dedicated space for storing transient data before uploading snapshots to the object storage service.

These transient data consist of the disk blocks that have changed since the previous snapshot of the volume.

The size of this storage space depends on the snapshot frequency and on how often the disk to be snapshotted is written.

The more frequently you snapshot a volume, the smaller the size of the changed blocks will be, so you require less space for transient snapshot data.

Similarly, if a volume is rarely written, there won’t be many changes between consecutive snapshots.

The size of this snapshot space determines the maximum amount of changed volume data Rok can snapshot.

This means that if, for example, the available Rok Snapshot Space is 200GiB, and you write 250GiB between two consecutive snapshots of a volume, then you won’t be able to snapshot the volume, unless you increase the size of the Rok Snapshot Space.

The amount of space reserved for snapshots is determined by the Rok Disk Manager based on the Disk Setup Script used for each cloud environment.

This guide will walk you through increasing the available Rok Snapshot Space to allow Rok to snapshot larger volumes that are frequently written or infrequently snapshotted.

Important

Note that any changes to the Disk Setup Script of Rok Disk Manager will affect only the available Rok Snapshot Space of new nodes. Currently, Rok Disk Manager doesn’t support resizing the Rok Snapshot Space of existing nodes.

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments

  2. Copy the disk setup script from the cloud provider overlay to the deploy overlay. Choose one of the following options, based on your cloud provider:

    AWS

    Azure

    Google Cloud

    root@rok-tools:~/ops/deployments# cp rok/rok-disk-manager/overlays/eks/patches/configmap.yaml rok/rok-disk-manager/overlays/deploy/patches/
    root@rok-tools:~/ops/deployments# cp rok/rok-disk-manager/overlays/aks/patches/configmap.yaml rok/rok-disk-manager/overlays/deploy/patches/
    root@rok-tools:~/ops/deployments# cp rok/rok-disk-manager/overlays/gke/patches/configmap.yaml rok/rok-disk-manager/overlays/deploy/patches/

  3. Edit rok/rok-disk-manager/overlays/deploy/patches/configmap.yaml and increase the size of the rok-fisks LV. By default, Rok uses 30% of the available disk space with a maximum of 200GiB. Increase that based on your needs, for example to 30% of the available disk space with a maximum of 700GiB:

    fiskslv_size = min(700 * GiB, 0.3 * rokvg.size); # <-- Edit this line to increase the maximum Rok Snapshot Size

  4. Edit rok/rok-disk-manager/overlays/deploy/kustomization.yaml and add the patch for the new disk setup script:

    # Apply patches patchesStrategicMerge: - patches/image-pull-secrets.yaml +- patches/configmap.yaml

  5. Track all changes in the git repository:

    root@rok-tools:~/ops/deployments# git add rok/rok-disk-manager

  6. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit -m "Increase Rok Snapshot Space"

  7. Apply the kustomization:

    root@rok-tools:~/ops/deployments# rok-deploy \ > --apply rok/rok-disk-manager/overlays/deploy

Verify

  1. From inside your management environment, verify that the Rok Disk Manager Disk Setup Script has been successfully updated. Check the size of the rok-fisks LV and verify that it has been updated to the new value:

    root@rok-tools:~# kubectl get cm -n rok-system disk-script -o jsonpath="{.data.disk-script}" ssds = get_disks(devices="/dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage*"); ebs = get_disks(devices="/dev/sd[f-p]"); md0 = raid("/dev/md0", bdevs=ssds + ebs, level=0); rokpv = pv(md0); rokvg = vg("rokvg", pvs=rokpv); fiskslv_size = min(700 * GiB, 0.3 * rokvg.size); fiskslv = lv(rokvg, "rok-fisks", size=fiskslv_size); filesystem = fs(fiskslv, "ext4"); mountpoint = mount(filesystem, "/mnt/data", persistent=False); _ = dir("/mnt/data/rok");

Summary

You have successfully increased the available Rok Snapshot Space for new nodes.

What’s Next

Check out the rest of the maintenance operations that you can perform on your cluster.