Create EKS Node Group

EKS supports both managed and self-managed node groups. You can use either of them based on your needs. For example, if you want to use a specific AMI then you should use self-managed node groups. For easier management via the Console prefer using managed node groups.

Both node group types use AutoScalingGroups (ASG) underneath, which, in turn, use a Launch Template that specifies the Kubernetes node instance configuration.

In this section we will see:

Local Storage requirements

Rok can run on any instance type, as long as there are disks available for it to use.

  • For instance types that have instance store volumes (local NVMe storage) attached, Rok will automatically find and use all of them.
  • For instance types without instance store volumes (EBS-only), you will need one or more extra EBS volumes of the exact same size. Rok will use all extra EBS volumes with devices names /dev/sd[f-p] (see also recommended device names for EBS volumes).

Important

Rok expects to use all the available extra volumes mentioned above exclusively.

Note

Since the size of EBS volumes affects their performance, we recommend that you use at least 500GiB per disk.

Note

You should not add extra EBS volumes to instances with instance store volumes. The default behavior for Rok is to use all available volumes as a RAID0 array, and this will lead to unpredictable performance.

Managed node groups

Amazon EKS managed node groups automate the provisioning and lifecycle management of nodes (Amazon EC2 instances) for Amazon EKS Kubernetes clusters.

  1. Specify the subnets to use. For the Auto Scaling Group, in contrast to the control plane, we can use a single AZ. The rule of thumb here is that the ASG minSize should be equal to or greater than the number of the Availability Zones it spans. Still, since we will make use of EBS volumes, we highly recommend that the ASG spans a single Availability zone. See Select subnets above for how to obtain the desired subnet IDs and make sure you set the SUBNETIDS environment variable accordingly.

  2. Choose an instance type of your preference. We recommend that you use an instance type that has instance store volumes (local NVMe storage) attached. For example, use m5d.4xlarge, which has 16CPU, 64G RAM, and 2x300 NVMe SSD:

    $ export INSTANCE_TYPE=m5d.4xlarge
    

    If you prefer to use an EBS-only instance type, e.g., m5.large, create the node group as mentioned below, and then, make sure to Add extra EBS volumes so that Rok can use them.

  3. Create the managed node group:

    $ aws eks create-nodegroup \
    >     --cluster-name ${CLUSTERNAME?} \
    >     --nodegroup-name general-workers \
    >     --disk-size 200 \
    >     --scaling-config minSize=1,maxSize=3,desiredSize=2 \
    >     --subnets ${SUBNETIDS?} \
    >     --instance-types ${INSTANCE_TYPE?} \
    >     --ami-type AL2_x86_64 \
    >     --node-role arn:aws:iam::${ACCOUNT_ID?}:role/eksWorkerNodeRole \
    >     --labels role=general-worker \
    >     --tags owner=${AWS_ACCOUNT?}/${AWS_IAM_USER?},kubernetes.io/cluster/${CLUSTERNAME?}=owned \
    >     --release-version 1.18.9-20210519 \
    >     --kubernetes-version 1.18
    

    Note

    Creating an EKS managed nodegroup with zero minimum size is currently not supported.

    Note

    You can create a Node Group for your cluster once its status is ACTIVE.

    Important

    The above command might fail with the following error message:

    “An error occurred (InvalidParameterException) when calling the CreateNodegroup operation: Requested Nodegroup release version X is invalid.”

    This means that Amazon has released a new AMI for EKS, most likely with an upgraded amazonlinux kernel. In this case, please coordinate with Arrikto’s Tech Team to ensure that container images that support the latest AMI and kernel are available.

  4. After a while the nodes of the managed nodegroup for your EKS cluster should be up and running. Verify with:

    $ aws eks describe-cluster --name ${CLUSTERNAME?} | jq .cluster.status
    "ACTIVE"
    

At this point, if you want allow other users to access your newly created EKS cluster you can follow the steps described in the (Optional) Share EKS cluster section.

Self-managed node groups

AWS CLI does not support creating self-managed node groups. To create one, follow the official instructions that use a CloudFormation template. Make sure to:

  1. Pick a name for the stack, e.g., <CLUSTERNAME>-workers

  2. Specify the name of the EKS cluster we created previously.

  3. For ClusterControlPlaneSecurityGroup, select the security group you created during the Create EKS Security Group section. In case you created a dedicated VPC, choose the SecurityGroups value from the AWS CloudFormation output:

    ../../../_images/cf-eks-vpc-output.png

    This is the security group to allow communication between your worker nodes and the Kubernetes control plane.

  4. Pick a name for the node group, e.g., general-workers. The created instances will be named after <CLUSTERNAME>-<NODEGROUPNAME>-Node.

  5. For the image use the latest one for 1.18 using the corresponding NodeImageIdSSMParam.

  6. Set a large enough NodeVolumeSize, e.g., 200 (GB) since this will hold Docker images and pod’s ephemeral storage.

  7. Select your desired instance type. If it is EBS-only see how to Add extra EBS volumes after node group creation.

  8. Select the VPC, and the subnets to spawn the workers in. Since we will use EBS volumes, we highly recommend that the ASG spans a single Availability zone. See Select subnets above for how to obtain the desired subnet IDs and make sure you choose them from the given drop down list.

  9. In Configure stack options, specify the Tags that Cluster Autoscaler requires so that it can discover the instances of the ASG automatically:

    Key Value
    k8s.io/cluster-autoscaler/enabled true
    k8s.io/cluster-autoscaler/<CLUSTERNAME> owned

Create the stack and wait for CloudFormation to

  • Create an IAM role that worker nodes will consume.
  • Create an AutoScalingGroup with a new Launch Template.
  • Create a security group that the worker nodes will use.
  • Modify given cluster security group to allow communication between control plane and worker nodes.

After the stack has finished creating, continue with the enable nodes to join your cluster section to complete the setup of the node group.

Add extra EBS volumes

In case you have used an EBS-only instance type for the node group, you will have to attach an extra EBS volume for Rok to use as local storage. To do that, you have to edit the Launch template that the underlying ASG is using. This is similar to what we do for EKS Upgrade. Specifically:

  1. Go to https://console.aws.amazon.com/ec2autoscaling/.

  2. Find the ASG associated with our node group.

  3. Edit its Launch template.

    ../../../_images/asg-edit-lt.png
  4. Create a new launch template version.

    ../../../_images/lt-new-version.png
  5. In the Storage (volumes) section click on Add new volume.

    ../../../_images/lt-add-new-volume.png
  6. Specify the specs of the extra EBS volume:

    • Size: 500 GiB
    • Device name: /dev/sdf
    • Volume type: gp2
    ../../../_images/lt-extra-ebs-volume.png
  7. Create the new template version.

    ../../../_images/lt-new-version-success.png
  8. Go back to the Launch template section of the ASG, refresh the drop down menu with the versions and select the newly created one.

    ../../../_images/asg-update-lt-version.png

From now on, all newly created instances will have an extra EBS disk. To replace the existing instances start an instance refresh operation by specifying zero for Minimum healthy percentage so that all instances are replaced at the same time.

Verify

  1. Verify that EC2 instances have been created:

    $ aws ec2 describe-instances \
    >     --filters Name=tag-key,Values=kubernetes.io/cluster/$CLUSTERNAME
    
  2. Verify that Kubernetes nodes have appeared:

    $ kubectl get nodes
    NAME                                         STATUS   ROLES    AGE    VERSION
    ip-172-31-0-86.us-west-2.compute.internal    Ready    <none>   8m2s   v1.18.9-eks-d1db3c
    ip-172-31-24-96.us-west-2.compute.internal   Ready    <none>   8m4s   v1.18.9-eks-d1db3c
    

What’s Next

The next step is to deploy Rok.