Increase Rok etcd EBS Volume IOPS

Rok uses etcd to persist metadata for your Rok volumes and snapshots. Rok etcd data is persisted in a PVC that is backed by an EBS volume. By default, this PVC uses the gp2 volume type. These volumes are SSD backed volumes that are suitable for general purpose usage.

However, when scaling Rok to hundreds of nodes, Rok etcd needs to handle a large amount of incoming traffic. In order to keep up, we highly recommend that you switch its volume to the io1 volume type, which is better suited for the high amount of IOPS required by large clusters.

This guide will walk you through converting the EBS volume behind the Rok etcd PVC from gp2 to io1.

Important

We strongly suggest that you convert the Rok etcd EBS volume to io1 in EKS clusters with 100 or more nodes.

What You’ll Need

Procedure

  1. Go to your rok-tools management environment and retrieve the name of the PersistentVolume Rok etcd uses:

    root@rok-tools:~# export ROK_ETCD_PV=$(kubectl get pvc \ > data-rok-etcd-0 -n rok -o jsonpath={.spec.volumeName})

  2. Retrieve the ID of the EBS volume Rok etcd uses:

    root@rok-tools:~# export ROK_ETCD_VOLUME_ID=$(kubectl get pv \ > -o jsonpath={.spec.awsElasticBlockStore.volumeID} $ROK_ETCD_PV)

  3. Extract the name of the EBS volume from its ID:

    root@rok-tools:~# export ROK_ETCD_VOLUME="${ROK_ETCD_VOLUME_ID##*/}"

  4. Change the type of the Rok etcd volume to io1, using the maximum allowed number of IOPS:

    root@rok-tools:~# aws ec2 modify-volume \ > --volume-id ${ROK_ETCD_VOLUME?} \ > --volume-type io1 \ > --iops 4000

    Note

    The maximum number of IOPS allowed for an io1 volume depends on its size and is 50 IOPS per Gigabyte. Since the Rok etcd volume is 80GiB, the maximum allowed number of IOPS is 4000.

    Troubleshooting

    Modification Rate Exceeded

    AWS only allows modifying an EBS volume once every 6 hours. If you have modified the same EBS volume less than 6 hours ago, the request will fail with the following message:

    An error occurred (VolumeModificationRateExceeded) when calling the ModifyVolume operation: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.

    In this case, wait until the 6 hours interval has passed and run the Procedure again.

Verify

  1. Go to your rok-tools management environment and retrieve the name of the PersistentVolume Rok etcd uses:

    root@rok-tools:~# export ROK_ETCD_PV=$(kubectl get pvc \ > data-rok-etcd-0 -n rok -o jsonpath={.spec.volumeName})

  2. Retrieve the ID of the EBS volume Rok etcd uses:

    root@rok-tools:~# export ROK_ETCD_VOLUME_ID=$(kubectl get pv \ > -o jsonpath={.spec.awsElasticBlockStore.volumeID} $ROK_ETCD_PV)

  3. Extract the name of the EBS volume from its ID:

    root@rok-tools:~# export ROK_ETCD_VOLUME="${ROK_ETCD_VOLUME_ID##*/}"

  4. Ensure that the type of the EBS volume is io1:

    root@rok-tools:~# aws ec2 describe-volumes --volume-ids $ROK_ETCD_VOLUME \ > --query 'Volumes[*].VolumeType' --output text io1

  5. Ensure that the number of IOPS of the volume is equal to 4000:

    root@rok-tools:~# aws ec2 describe-volumes --volume-ids $ROK_ETCD_VOLUME \ > --query 'Volumes[*].Iops' --output text 4000

Summary

You have successfully converted the type of the EBS volume Rok etcd uses to io1.

What’s Next

Check out the rest of the maintenance operations that you can perform on your cluster.

PreviousNext