Create EKS Self-Managed Node Group

This section will guide you through creating a self-managed node group.

What You’ll Need

Procedure

  1. Download the CloudFormation stack for self-managed node groups below:

    amazon-eks-nodegroup.yaml
    1AWSTemplateFormatVersion: "2010-09-09"
    2
    3Description: Amazon EKS - Node Group
    4-674
    4
    5Metadata:
    6 "AWS::CloudFormation::Interface":
    7 ParameterGroups:
    8 - Label:
    9 default: EKS Cluster
    10 Parameters:
    11 - ClusterName
    12 - ClusterControlPlaneSecurityGroup
    13 - Label:
    14 default: Worker Node Configuration
    15 Parameters:
    16 - NodeGroupName
    17 - NodeAutoScalingGroupMinSize
    18 - NodeAutoScalingGroupDesiredCapacity
    19 - NodeAutoScalingGroupMaxSize
    20 - NodeInstanceType
    21 - NodeImageIdSSMParam
    22 - NodeImageId
    23 - NodeVolumeSize
    24 - NodeExtraEBSVolumeSize
    25 - KeyName
    26 - BootstrapArguments
    27 - DisableIMDSv1
    28 - Label:
    29 default: Worker Network Configuration
    30 Parameters:
    31 - VpcId
    32 - Subnets
    33
    34Parameters:
    35 BootstrapArguments:
    36 Type: String
    37 Default: ""
    38 Description: "Arguments to pass to the bootstrap script. See files/bootstrap.sh in https://github.com/awslabs/amazon-eks-ami"
    39
    40 ClusterControlPlaneSecurityGroup:
    41 Type: "AWS::EC2::SecurityGroup::Id"
    42 Description: The security group of the cluster control plane.
    43
    44 ClusterName:
    45 Type: String
    46 Description: The cluster name provided when the cluster was created. If it is incorrect, nodes will not be able to join the cluster.
    47
    48 KeyName:
    49 Type: "AWS::EC2::KeyPair::KeyName"
    50 Description: The EC2 Key Pair to allow SSH access to the instances
    51
    52 NodeAutoScalingGroupDesiredCapacity:
    53 Type: Number
    54 Default: 3
    55 Description: Desired capacity of Node Group ASG.
    56
    57 NodeAutoScalingGroupMaxSize:
    58 Type: Number
    59 Default: 4
    60 Description: Maximum size of Node Group ASG. Set to at least 1 greater than NodeAutoScalingGroupDesiredCapacity.
    61
    62 NodeAutoScalingGroupMinSize:
    63 Type: Number
    64 Default: 1
    65 Description: Minimum size of Node Group ASG.
    66
    67 NodeGroupName:
    68 Type: String
    69 Description: Unique identifier for the Node Group.
    70
    71 NodeImageId:
    72 Type: String
    73 Default: ""
    74 Description: (Optional) Specify your own custom image ID. This value overrides any AWS Systems Manager Parameter Store value specified above.
    75
    76 NodeImageIdSSMParam:
    77 Type: "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>"
    78 Default: /aws/service/eks/optimized-ami/1.17/amazon-linux-2/recommended/image_id
    79 Description: AWS Systems Manager Parameter Store parameter of the AMI ID for the worker node instances. Change this value to match the version of Kubernetes you are using.
    80
    81 DisableIMDSv1:
    82 Type: String
    83 Default: "true"
    84 AllowedValues:
    85 - "false"
    86 - "true"
    87
    88 NodeInstanceType:
    89 Type: String
    90 Default: t3.medium
    91 AllowedValues:
    92 - a1.2xlarge
    93 - a1.4xlarge
    94 - a1.large
    95 - a1.medium
    96 - a1.metal
    97 - a1.xlarge
    98 - c1.medium
    99 - c1.xlarge
    100 - c3.2xlarge
    101 - c3.4xlarge
    102 - c3.8xlarge
    103 - c3.large
    104 - c3.xlarge
    105 - c4.2xlarge
    106 - c4.4xlarge
    107 - c4.8xlarge
    108 - c4.large
    109 - c4.xlarge
    110 - c5.12xlarge
    111 - c5.18xlarge
    112 - c5.24xlarge
    113 - c5.2xlarge
    114 - c5.4xlarge
    115 - c5.9xlarge
    116 - c5.large
    117 - c5.metal
    118 - c5.xlarge
    119 - c5a.12xlarge
    120 - c5a.16xlarge
    121 - c5a.24xlarge
    122 - c5a.2xlarge
    123 - c5a.4xlarge
    124 - c5a.8xlarge
    125 - c5a.large
    126 - c5a.metal
    127 - c5a.xlarge
    128 - c5ad.12xlarge
    129 - c5ad.16xlarge
    130 - c5ad.24xlarge
    131 - c5ad.2xlarge
    132 - c5ad.4xlarge
    133 - c5ad.8xlarge
    134 - c5ad.large
    135 - c5ad.metal
    136 - c5ad.xlarge
    137 - c5d.12xlarge
    138 - c5d.18xlarge
    139 - c5d.24xlarge
    140 - c5d.2xlarge
    141 - c5d.4xlarge
    142 - c5d.9xlarge
    143 - c5d.large
    144 - c5d.metal
    145 - c5d.xlarge
    146 - c5n.18xlarge
    147 - c5n.2xlarge
    148 - c5n.4xlarge
    149 - c5n.9xlarge
    150 - c5n.large
    151 - c5n.metal
    152 - c5n.xlarge
    153 - c6g.12xlarge
    154 - c6g.16xlarge
    155 - c6g.2xlarge
    156 - c6g.4xlarge
    157 - c6g.8xlarge
    158 - c6g.large
    159 - c6g.medium
    160 - c6g.metal
    161 - c6g.xlarge
    162 - c6gd.12xlarge
    163 - c6gd.16xlarge
    164 - c6gd.2xlarge
    165 - c6gd.4xlarge
    166 - c6gd.8xlarge
    167 - c6gd.large
    168 - c6gd.medium
    169 - c6gd.metal
    170 - c6gd.xlarge
    171 - cc2.8xlarge
    172 - cr1.8xlarge
    173 - d2.2xlarge
    174 - d2.4xlarge
    175 - d2.8xlarge
    176 - d2.xlarge
    177 - f1.16xlarge
    178 - f1.2xlarge
    179 - f1.4xlarge
    180 - g2.2xlarge
    181 - g2.8xlarge
    182 - g3.16xlarge
    183 - g3.4xlarge
    184 - g3.8xlarge
    185 - g3s.xlarge
    186 - g4dn.12xlarge
    187 - g4dn.16xlarge
    188 - g4dn.2xlarge
    189 - g4dn.4xlarge
    190 - g4dn.8xlarge
    191 - g4dn.metal
    192 - g4dn.xlarge
    193 - h1.16xlarge
    194 - h1.2xlarge
    195 - h1.4xlarge
    196 - h1.8xlarge
    197 - hs1.8xlarge
    198 - i2.2xlarge
    199 - i2.4xlarge
    200 - i2.8xlarge
    201 - i2.xlarge
    202 - i3.16xlarge
    203 - i3.2xlarge
    204 - i3.4xlarge
    205 - i3.8xlarge
    206 - i3.large
    207 - i3.metal
    208 - i3.xlarge
    209 - i3en.12xlarge
    210 - i3en.24xlarge
    211 - i3en.2xlarge
    212 - i3en.3xlarge
    213 - i3en.6xlarge
    214 - i3en.large
    215 - i3en.metal
    216 - i3en.xlarge
    217 - inf1.24xlarge
    218 - inf1.2xlarge
    219 - inf1.6xlarge
    220 - inf1.xlarge
    221 - m1.large
    222 - m1.medium
    223 - m1.small
    224 - m1.xlarge
    225 - m2.2xlarge
    226 - m2.4xlarge
    227 - m2.xlarge
    228 - m3.2xlarge
    229 - m3.large
    230 - m3.medium
    231 - m3.xlarge
    232 - m4.10xlarge
    233 - m4.16xlarge
    234 - m4.2xlarge
    235 - m4.4xlarge
    236 - m4.large
    237 - m4.xlarge
    238 - m5.12xlarge
    239 - m5.16xlarge
    240 - m5.24xlarge
    241 - m5.2xlarge
    242 - m5.4xlarge
    243 - m5.8xlarge
    244 - m5.large
    245 - m5.metal
    246 - m5.xlarge
    247 - m5a.12xlarge
    248 - m5a.16xlarge
    249 - m5a.24xlarge
    250 - m5a.2xlarge
    251 - m5a.4xlarge
    252 - m5a.8xlarge
    253 - m5a.large
    254 - m5a.xlarge
    255 - m5ad.12xlarge
    256 - m5ad.16xlarge
    257 - m5ad.24xlarge
    258 - m5ad.2xlarge
    259 - m5ad.4xlarge
    260 - m5ad.8xlarge
    261 - m5ad.large
    262 - m5ad.xlarge
    263 - m5d.12xlarge
    264 - m5d.16xlarge
    265 - m5d.24xlarge
    266 - m5d.2xlarge
    267 - m5d.4xlarge
    268 - m5d.8xlarge
    269 - m5d.large
    270 - m5d.metal
    271 - m5d.xlarge
    272 - m5dn.12xlarge
    273 - m5dn.16xlarge
    274 - m5dn.24xlarge
    275 - m5dn.2xlarge
    276 - m5dn.4xlarge
    277 - m5dn.8xlarge
    278 - m5dn.large
    279 - m5dn.xlarge
    280 - m5n.12xlarge
    281 - m5n.16xlarge
    282 - m5n.24xlarge
    283 - m5n.2xlarge
    284 - m5n.4xlarge
    285 - m5n.8xlarge
    286 - m5n.large
    287 - m5n.xlarge
    288 - m6g.12xlarge
    289 - m6g.16xlarge
    290 - m6g.2xlarge
    291 - m6g.4xlarge
    292 - m6g.8xlarge
    293 - m6g.large
    294 - m6g.medium
    295 - m6g.metal
    296 - m6g.xlarge
    297 - m6gd.12xlarge
    298 - m6gd.16xlarge
    299 - m6gd.2xlarge
    300 - m6gd.4xlarge
    301 - m6gd.8xlarge
    302 - m6gd.large
    303 - m6gd.medium
    304 - m6gd.metal
    305 - m6gd.xlarge
    306 - p2.16xlarge
    307 - p2.8xlarge
    308 - p2.xlarge
    309 - p3.16xlarge
    310 - p3.2xlarge
    311 - p3.8xlarge
    312 - p3dn.24xlarge
    313 - p4d.24xlarge
    314 - r3.2xlarge
    315 - r3.4xlarge
    316 - r3.8xlarge
    317 - r3.large
    318 - r3.xlarge
    319 - r4.16xlarge
    320 - r4.2xlarge
    321 - r4.4xlarge
    322 - r4.8xlarge
    323 - r4.large
    324 - r4.xlarge
    325 - r5.12xlarge
    326 - r5.16xlarge
    327 - r5.24xlarge
    328 - r5.2xlarge
    329 - r5.4xlarge
    330 - r5.8xlarge
    331 - r5.large
    332 - r5.metal
    333 - r5.xlarge
    334 - r5a.12xlarge
    335 - r5a.16xlarge
    336 - r5a.24xlarge
    337 - r5a.2xlarge
    338 - r5a.4xlarge
    339 - r5a.8xlarge
    340 - r5a.large
    341 - r5a.xlarge
    342 - r5ad.12xlarge
    343 - r5ad.16xlarge
    344 - r5ad.24xlarge
    345 - r5ad.2xlarge
    346 - r5ad.4xlarge
    347 - r5ad.8xlarge
    348 - r5ad.large
    349 - r5ad.xlarge
    350 - r5d.12xlarge
    351 - r5d.16xlarge
    352 - r5d.24xlarge
    353 - r5d.2xlarge
    354 - r5d.4xlarge
    355 - r5d.8xlarge
    356 - r5d.large
    357 - r5d.metal
    358 - r5d.xlarge
    359 - r5dn.12xlarge
    360 - r5dn.16xlarge
    361 - r5dn.24xlarge
    362 - r5dn.2xlarge
    363 - r5dn.4xlarge
    364 - r5dn.8xlarge
    365 - r5dn.large
    366 - r5dn.xlarge
    367 - r5n.12xlarge
    368 - r5n.16xlarge
    369 - r5n.24xlarge
    370 - r5n.2xlarge
    371 - r5n.4xlarge
    372 - r5n.8xlarge
    373 - r5n.large
    374 - r5n.xlarge
    375 - r6g.12xlarge
    376 - r6g.16xlarge
    377 - r6g.2xlarge
    378 - r6g.4xlarge
    379 - r6g.8xlarge
    380 - r6g.large
    381 - r6g.medium
    382 - r6g.metal
    383 - r6g.xlarge
    384 - r6gd.12xlarge
    385 - r6gd.16xlarge
    386 - r6gd.2xlarge
    387 - r6gd.4xlarge
    388 - r6gd.8xlarge
    389 - r6gd.large
    390 - r6gd.medium
    391 - r6gd.metal
    392 - r6gd.xlarge
    393 - t1.micro
    394 - t2.2xlarge
    395 - t2.large
    396 - t2.medium
    397 - t2.micro
    398 - t2.nano
    399 - t2.small
    400 - t2.xlarge
    401 - t3.2xlarge
    402 - t3.large
    403 - t3.medium
    404 - t3.micro
    405 - t3.nano
    406 - t3.small
    407 - t3.xlarge
    408 - t3a.2xlarge
    409 - t3a.large
    410 - t3a.medium
    411 - t3a.micro
    412 - t3a.nano
    413 - t3a.small
    414 - t3a.xlarge
    415 - t4g.2xlarge
    416 - t4g.large
    417 - t4g.medium
    418 - t4g.micro
    419 - t4g.nano
    420 - t4g.small
    421 - t4g.xlarge
    422 - u-12tb1.metal
    423 - u-18tb1.metal
    424 - u-24tb1.metal
    425 - u-6tb1.metal
    426 - u-9tb1.metal
    427 - x1.16xlarge
    428 - x1.32xlarge
    429 - x1e.16xlarge
    430 - x1e.2xlarge
    431 - x1e.32xlarge
    432 - x1e.4xlarge
    433 - x1e.8xlarge
    434 - x1e.xlarge
    435 - z1d.12xlarge
    436 - z1d.2xlarge
    437 - z1d.3xlarge
    438 - z1d.6xlarge
    439 - z1d.large
    440 - z1d.metal
    441 - z1d.xlarge
    442 ConstraintDescription: Must be a valid EC2 instance type
    443 Description: EC2 instance type for the node instances
    444
    445 NodeVolumeSize:
    446 Type: Number
    447 Default: 20
    448 Description: Node volume size
    449
    450 NodeExtraEBSVolumeSize:
    451 Type: Number
    452 Default: 0
    453 Description: Extra EBS volume size
    454
    455 Subnets:
    456 Type: "List<AWS::EC2::Subnet::Id>"
    457 Description: The subnets where workers can be created.
    458
    459 VpcId:
    460 Type: "AWS::EC2::VPC::Id"
    461 Description: The VPC of the worker instances
    462
    463Mappings:
    464 PartitionMap:
    465 aws:
    466 EC2ServicePrincipal: "ec2.amazonaws.com"
    467 aws-us-gov:
    468 EC2ServicePrincipal: "ec2.amazonaws.com"
    469 aws-cn:
    470 EC2ServicePrincipal: "ec2.amazonaws.com.cn"
    471 aws-iso:
    472 EC2ServicePrincipal: "ec2.c2s.ic.gov"
    473 aws-iso-b:
    474 EC2ServicePrincipal: "ec2.sc2s.sgov.gov"
    475
    476Conditions:
    477 HasNodeImageId: !Not
    478 - "Fn::Equals":
    479 - !Ref NodeImageId
    480 - ""
    481
    482 IMDSv1Disabled:
    483 "Fn::Equals":
    484 - !Ref DisableIMDSv1
    485 - "true"
    486
    487 HasNodeExtraDisk: !Not
    488 - "Fn::Equals":
    489 - !Ref NodeExtraEBSVolumeSize
    490 - 0
    491
    492
    493Resources:
    494 NodeInstanceRole:
    495 Type: "AWS::IAM::Role"
    496 Properties:
    497 AssumeRolePolicyDocument:
    498 Version: "2012-10-17"
    499 Statement:
    500 - Effect: Allow
    501 Principal:
    502 Service:
    503 - !FindInMap [PartitionMap, !Ref "AWS::Partition", EC2ServicePrincipal]
    504 Action:
    505 - "sts:AssumeRole"
    506 ManagedPolicyArns:
    507 - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy"
    508 - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy"
    509 - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
    510 Path: /
    511
    512 NodeInstanceProfile:
    513 Type: "AWS::IAM::InstanceProfile"
    514 Properties:
    515 Path: /
    516 Roles:
    517 - !Ref NodeInstanceRole
    518
    519 NodeSecurityGroup:
    520 Type: "AWS::EC2::SecurityGroup"
    521 Properties:
    522 GroupDescription: Security group for all nodes in the cluster
    523 Tags:
    524 - Key: !Sub kubernetes.io/cluster/${ClusterName}
    525 Value: owned
    526 VpcId: !Ref VpcId
    527
    528 NodeSecurityGroupIngress:
    529 Type: "AWS::EC2::SecurityGroupIngress"
    530 DependsOn: NodeSecurityGroup
    531 Properties:
    532 Description: Allow node to communicate with each other
    533 FromPort: 0
    534 GroupId: !Ref NodeSecurityGroup
    535 IpProtocol: "-1"
    536 SourceSecurityGroupId: !Ref NodeSecurityGroup
    537 ToPort: 65535
    538
    539 ClusterControlPlaneSecurityGroupIngress:
    540 Type: "AWS::EC2::SecurityGroupIngress"
    541 DependsOn: NodeSecurityGroup
    542 Properties:
    543 Description: Allow pods to communicate with the cluster API Server
    544 FromPort: 443
    545 GroupId: !Ref ClusterControlPlaneSecurityGroup
    546 IpProtocol: tcp
    547 SourceSecurityGroupId: !Ref NodeSecurityGroup
    548 ToPort: 443
    549
    550 ControlPlaneEgressToNodeSecurityGroup:
    551 Type: "AWS::EC2::SecurityGroupEgress"
    552 DependsOn: NodeSecurityGroup
    553 Properties:
    554 Description: Allow the cluster control plane to communicate with worker Kubelet and pods
    555 DestinationSecurityGroupId: !Ref NodeSecurityGroup
    556 FromPort: 1025
    557 GroupId: !Ref ClusterControlPlaneSecurityGroup
    558 IpProtocol: tcp
    559 ToPort: 65535
    560
    561 ControlPlaneEgressToNodeSecurityGroupOn443:
    562 Type: "AWS::EC2::SecurityGroupEgress"
    563 DependsOn: NodeSecurityGroup
    564 Properties:
    565 Description: Allow the cluster control plane to communicate with pods running extension API servers on port 443
    566 DestinationSecurityGroupId: !Ref NodeSecurityGroup
    567 FromPort: 443
    568 GroupId: !Ref ClusterControlPlaneSecurityGroup
    569 IpProtocol: tcp
    570 ToPort: 443
    571
    572 NodeSecurityGroupFromControlPlaneIngress:
    573 Type: "AWS::EC2::SecurityGroupIngress"
    574 DependsOn: NodeSecurityGroup
    575 Properties:
    576 Description: Allow worker Kubelets and pods to receive communication from the cluster control plane
    577 FromPort: 1025
    578 GroupId: !Ref NodeSecurityGroup
    579 IpProtocol: tcp
    580 SourceSecurityGroupId: !Ref ClusterControlPlaneSecurityGroup
    581 ToPort: 65535
    582
    583 NodeSecurityGroupFromControlPlaneOn443Ingress:
    584 Type: "AWS::EC2::SecurityGroupIngress"
    585 DependsOn: NodeSecurityGroup
    586 Properties:
    587 Description: Allow pods running extension API servers on port 443 to receive communication from cluster control plane
    588 FromPort: 443
    589 GroupId: !Ref NodeSecurityGroup
    590 IpProtocol: tcp
    591 SourceSecurityGroupId: !Ref ClusterControlPlaneSecurityGroup
    592 ToPort: 443
    593
    594 NodeLaunchTemplate:
    595 Type: "AWS::EC2::LaunchTemplate"
    596 Properties:
    597 LaunchTemplateData:
    598 BlockDeviceMappings: !If
    599 - HasNodeExtraDisk
    600 - - DeviceName: /dev/xvda
    601 Ebs:
    602 DeleteOnTermination: true
    603 VolumeSize: !Ref NodeVolumeSize
    604 VolumeType: gp2
    605 - DeviceName: /dev/sdf
    606 Ebs:
    607 DeleteOnTermination: true
    608 VolumeSize: !Ref NodeExtraEBSVolumeSize
    609 VolumeType: gp2
    610 - - DeviceName: /dev/xvda
    611 Ebs:
    612 DeleteOnTermination: true
    613 VolumeSize: !Ref NodeVolumeSize
    614 VolumeType: gp2
    615
    616 IamInstanceProfile:
    617 Arn: !GetAtt NodeInstanceProfile.Arn
    618 ImageId: !If
    619 - HasNodeImageId
    620 - !Ref NodeImageId
    621 - !Ref NodeImageIdSSMParam
    622 InstanceType: !Ref NodeInstanceType
    623 KeyName: !Ref KeyName
    624 SecurityGroupIds:
    625 - !Ref NodeSecurityGroup
    626 UserData: !Base64
    627 "Fn::Sub": |
    628 #!/bin/bash
    629 set -o xtrace
    630 /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
    631 /opt/aws/bin/cfn-signal --exit-code $? \
    632 --stack ${AWS::StackName} \
    633 --resource NodeGroup \
    634 --region ${AWS::Region}
    635 MetadataOptions:
    636 HttpPutResponseHopLimit : 1
    637 HttpEndpoint: enabled
    638 HttpTokens: !If
    639 - IMDSv1Disabled
    640 - required
    641 - optional
    642
    643 NodeGroup:
    644 Type: "AWS::AutoScaling::AutoScalingGroup"
    645 Properties:
    646 DesiredCapacity: !Ref NodeAutoScalingGroupDesiredCapacity
    647 LaunchTemplate:
    648 LaunchTemplateId: !Ref NodeLaunchTemplate
    649 Version: !GetAtt NodeLaunchTemplate.LatestVersionNumber
    650 MaxSize: !Ref NodeAutoScalingGroupMaxSize
    651 MinSize: !Ref NodeAutoScalingGroupMinSize
    652 Tags:
    653 - Key: Name
    654 PropagateAtLaunch: true
    655 Value: !Sub ${ClusterName}-${NodeGroupName}-Node
    656 - Key: !Sub kubernetes.io/cluster/${ClusterName}
    657 PropagateAtLaunch: true
    658 Value: owned
    659 VPCZoneIdentifier: !Ref Subnets
    660 UpdatePolicy:
    661 AutoScalingRollingUpdate:
    662 MaxBatchSize: 1
    663 MinInstancesInService: !Ref NodeAutoScalingGroupDesiredCapacity
    664 PauseTime: PT5M
    665
    666Outputs:
    667 NodeInstanceRole:
    668 Description: The node instance role
    669 Value: !GetAtt NodeInstanceRole.Arn
    670
    671 NodeSecurityGroup:
    672 Description: The security group for the node group
    673 Value: !Ref NodeSecurityGroup
    674
    675 NodeAutoScalingGroup:
    676 Description: The autoscaling group
    677 Value: !Ref NodeGroup
  2. Go to AWS CloudFormation Console.

  3. Click the Create stack dropdown menu and select with new resources(standard).

  4. In Specify template choose Upload a template file and upload the amazon-eks-nodegroup.yaml file you downloaded in the first step. Then click Next.

  5. In section Stack name pick a name for the stack, for example, <EKS_CLUSTER>-workers, where <EKS_CLUSTER> is the name of your cluster.

  6. In section Parameters specify the name of the EKS cluster you created previously.

  7. For ClusterControlPlaneSecurityGroup, select the security group created with your VPC CloudFormation stack. To find it, go to AWS CloudFormation Console, select the CF stack of your VPC, go to Outputs, and note down the SecurityGroups value:

    ../../../../_images/cf-eks-vpc-output.png

    This is the security group to allow communication between your worker nodes and the Kubernetes control plane.

  8. Pick a name for the node group, for example, general-workers. The created instances will be named after <EKS_CLUSTER>-<NODEGROUPNAME>-Node.

  9. In NodeInstanceType select your desired instance type. We recommend that you use an instance type that has instance store volumes (local NVMe storage) attached, such as m5d.4xlarge.

  10. Set NodeImageIdSSMParam by choosing one of the following options based on the Kubernetes version of your cluster.

    /aws/service/eks/optimized-ami/1.22/amazon-linux-2/amazon-eks-node-1.22-v20220914/image_id
    /aws/service/eks/optimized-ami/1.21/amazon-linux-2/amazon-eks-node-1.21-v20220914/image_id
  11. Set a large enough NodeVolumeSize, for example, 200 (GB) since this will hold Docker images and ephemeral storage of Pods.

  12. If the instance type you chose is EBS-only (that is, it doesn’t have ephemeral storage attached), set a large enough NodeExtraEBSVolumeSize, for example, 500 (GB) since this will hold your Rok storage.

  13. Under KeyName, specify an EC2 key pair to allow SSH access to the instances. This option is required by the CloudFormation template that we are using.

  14. Ensure that DisableIMDSv1 is set to true so that worker nodes use IMDSv2 only.

  15. Select the VPC, and the subnets to spawn the workers in. If you use EBS volumes, we highly recommend that the Auto Scaling group (ASG) spans a single Availability zone. Choose them from the given drop down list. Then click Next.

  16. In Configure stack options, add the following Tags that Cluster Autoscaler requires so that it can discover the instances of the ASG automatically. Then click Next.

    Key Value
    k8s.io/cluster-autoscaler/enabled true
    k8s.io/cluster-autoscaler/<EKS_CLUSTER> owned
  17. Review your CloudFormation stack to make sure you have configured it as instructed above. Check the necessary boxes and click Create stack.

    Note

    CloudFormation will

    • Create an IAM role that worker nodes will consume.
    • Create an AutoScalingGroup with a new Launch Template.
    • Create a security group that the worker nodes will use.
    • Modify given cluster security group to allow communication between control plane and worker nodes.
  18. After the stack has finished creating, you need to allow the nodes to join the cluster. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  19. Specify the CloudFormation stack name you used for creating the node group:

    root@rok-tools:~/ops/deployments# export CF_STACK_NAME=<CF_STACK_NAME>

    Replace <CF_STACK_NAME> with the your node group’s CF stack, for example:

    root@rok-tools:~/ops/deployments# export CF_STACK_NAME=arrikto-cluster-workers
  20. Obtain the IAM role that CloudFormation created for the nodes to run with:

    root@rok-tools:~/ops/deployments# aws cloudformation describe-stacks \ > --stack-name ${CF_STACK_NAME?} \ > --query 'Stacks[].Outputs[?OutputKey==`NodeInstanceRole`].OutputValue' \ > --output text arn:aws:iam::123456789012:role/arrikto-cluster-workers-NodeInstanceRole-1UR1Z9YTC2TTK
  21. Inspect the aws-auth ConfigMap to get existing entries if any:

    root@rok-tools:~/ops/deployments# kubectl get configmap aws-auth \ > -n kube-system -o jsonpath={.data.mapRoles}
  22. Edit rok/eks/aws-auth.yaml and add a new entry under mapRoles, including the entries you found in previous step:

    data: mapRoles: | - groups: - system:bootstrappers - system:nodes rolearn: arn:aws:iam::123456789012:role/arrikto-cluster-workers-NodeInstanceRole-1UR1Z9YTC2TTK username: system:node:{{EC2PrivateDNSName}}
  23. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit \ > -am "Create EKS Self-managed Node Group: Allow $CF_STACK_NAME to join the cluster"
  24. Apply the manifest:

    root@rok-tools:~/ops/deployments# kubectl apply -f rok/eks/aws-auth.yaml

Verify

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Restore the required context from previous sections:

    root@rok-tools:~/ops/deployments# source <(cat deploy/env.eks-cluster)
    root@rok-tools:~/ops/deployments# export EKS_CLUSTER
  3. Verify that EC2 instances have been created:

    root@rok-tools:~/ops/deployments# aws ec2 describe-instances \ > --filters Name=tag-key,Values=kubernetes.io/cluster/${EKS_CLUSTER?} { "Reservations": [ { "Groups": [], "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-012b81faa674369fc", "InstanceId": "i-0a1795ed2c92c16d5", "InstanceType": "m5.large", "LaunchTime": "2021-07-27T08:39:41+00:00", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "eu-central-1b", "GroupName": "", "Tenancy": "default" }, ...
  4. Verify that all EC2 instances use IMDSv2 only:

    1. Retrieve the EC2 instance IDs of your cluster:

      root@rok-tools:~/ops/deployments# aws ec2 describe-instances \ > --filters Name=tag:kubernetes.io/cluster/${EKS_CLUSTER?},Values=owned \ > --query "Reservations[*].Instances[*].InstanceId" --output text) i-075363bbf64a60e04 i-06a5ee72c6eed1bad
    2. Repeat the steps below for each one of the EC2 instance IDs in the list of the previous step.

      1. Specify the ID of the EC2 instance to operate on:

        root@rok-tools:~/ops/deployments# export INSTANCE_ID=<INSTANCE_ID>

        Replace <INSTANCE_ID> with one of the IDs you found in the previous step, for example:

        root@rok-tools:~/ops/deployments# export INSTANCE_ID=i-075363bbf64a60e04
      2. Verify that the HttpTokens metadata option is set to required:

        root@rok-tools:~/ops/deployments# aws ec2 get-launch-template-data \ > --instance-id ${INSTANCE_ID?} \ > --query 'LaunchTemplateData.MetadataOptions.HttpTokens == `required`' true
      3. Verify that the HttpPutResponseHopLimit metadata option is set to 1:

        root@rok-tools:~/ops/deployments# aws ec2 get-launch-template-data \ > --instance-id ${INSTANCE_ID?} \ > --query 'LaunchTemplateData.MetadataOptions.HttpPutResponseHopLimit == `1`' true
      4. Go back to step i and repeat steps i-iv for the remaining instance IDs.

  5. Verify that Kubernetes nodes have appeared:

    root@rok-tools:~/ops/deployments# kubectl get nodes NAME STATUS ROLES AGE VERSION ip-172-31-0-86.us-west-2.compute.internal Ready <none> 8m2s v1.22.12-eks-ba74326 ip-172-31-24-96.us-west-2.compute.internal Ready <none> 8m4s v1.22.12-eks-ba74326

Summary

You have successfully created a self-managed node group.

What’s Next

The next step to disable unsafe operations for your EKS cluster.