Create IAM Role for Cluster Autoscaler

This section will guide you through creating an IAM role for the Cluster Autoscaler on your EKS cluster.

See also

Choose one of the following options to create an IAM role for your Cluster Autoscaler:

Check Your Environment

To create this role, you are going to deploy a CloudFormation stack. When working with AWS CloudFormation stacks to manage resources, not only do you need sufficient permissions on AWS CloudFormation, but also on the underlying resources that are defined in the template.

In order to create an IAM role with proper IAM policies attached to it for your EKS Cluster Autoscaler using AWS CloudFormation you need permissions for the following actions:

  • Deploy AWS CloudFormation stacks.
  • Create IAM roles.
  • Create IAM policies.
  • Attach managed IAM policies to IAM roles.

Note

If you do not have the above permissions, contact your AWS administrator to grant sufficient permissions to your IAM user or deploy the below AWS CloudFormation stack for you.

Option 1: Create IAM Role for Cluster Autoscaler Automatically (preferred)

Create an IAM role for your Cluster Autoscaler by following the on-screen instructions on the rok-deploy user interface.

If rok-deploy is not already running, start it with:

root@rok-tools:~# rok-deploy --run-from autoscaler-eks-iam
../../../_images/autoscaler-eks-iam.png

Proceed to the Summary section.

Option 2: Create IAM Role for Cluster Autoscaler Manually

If you want to create an IAM role your for Cluster Autoscaler manually, follow the instructions below.

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Restore the required context from previous sections:

    root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,eks-cluster,\ > eks-identity})
    root@rok-tools:~/ops/deployments# export AWS_ACCOUNT_ID AWS_DEFAULT_REGION \ > EKS_CLUSTER_OIDC EKS_CLUSTER
  3. Set the name of the IAM role for the Cluster Autoscaler:

    root@rok-tools:~/ops/deployments# export AUTOSCALER_EKS_IAM_ROLE=rok-\ > ${AWS_DEFAULT_REGION?}-${EKS_CLUSTER?}-cluster-autoscaler
  4. Verify that the IAM role name you specified is not longer than 64 characters:

    root@rok-tools:~/ops/deployments# [[ ${#AUTOSCALER_EKS_IAM_ROLE} -le 64 ]] && echo OK || echo FAIL OK

    Troubleshooting

    The output of the command is FAIL

    Go back to step 3 and specify a shorter name. Ensure the new name is not already in use.

  5. Set the name of the IAM policy for the Cluster Autoscaler:

    root@rok-tools:~/ops/deployments# export AUTOSCALER_EKS_IAM_POLICY=rok-${AWS_DEFAULT_REGION?}-${EKS_CLUSTER?}-cluster-autoscaler
  6. Verify that the IAM policy name you specified is not longer than 128 characters:

    root@rok-tools:~/ops/deployments# [[ ${#AUTOSCALER_EKS_IAM_POLICY} -le 128 ]] && echo OK || echo FAIL OK

    Troubleshooting

    The output of the command is FAIL

    Go back to step 5 and specify a shorter name. Ensure the new name is not already in use.

  7. Set the name of the CloudFormation stack you will deploy:

    root@rok-tools:~/ops/deployments# export AUTOSCALER_EKS_IAM_CF=rok-${AWS_DEFAULT_REGION?}-${EKS_CLUSTER?}-cluster-autoscaler
  8. Verify that the CloudFormation stack name you specified is not longer than 128 characters:

    root@rok-tools:~/ops/deployments# [[ ${#AUTOSCALER_EKS_IAM_CF} -le 128 ]] && echo OK || echo FAIL OK

    Troubleshooting

    The output of the command is FAIL

    Go back to step 7 and specify a shorter name. Ensure the new name is not already in use.

  9. Generate the AWS CloudFormation stack:

    root@rok-tools:~/ops/deployments# j2 rok/eks/autoscaler-eks-iam-resources.yaml.j2 \ > -o rok/eks/autoscaler-eks-iam-resources.yaml

    Alternatively, download the autoscaler-eks-iam-resources CloudFormation template provided below and use it locally.

    autoscaler-eks-iam-resources.yaml.j2
    1Metadata:
    2 Rok::StackName: {{ AUTOSCALER_EKS_IAM_CF }}
    3
    4-36
    4Resources:
    5 ClusterAutoscalerRole:
    6 Type: AWS::IAM::Role
    7 Description: Cluster Autoscaler Role
    8 Properties:
    9 RoleName: {{ AUTOSCALER_EKS_IAM_ROLE }}
    10 AssumeRolePolicyDocument:
    11 Version: '2012-10-17'
    12 Statement:
    13 - Effect: Allow
    14 Action: sts:AssumeRoleWithWebIdentity
    15 Principal:
    16 Federated: arn:aws:iam::{{ AWS_ACCOUNT_ID }}:oidc-provider/{{ EKS_CLUSTER_OIDC }}
    17 Condition:
    18 StringEquals:
    19 {{ EKS_CLUSTER_OIDC }}:sub: system:serviceaccount:kube-system:cluster-autoscaler
    20 ManagedPolicyArns:
    21 - Ref: ClusterAutoscalerPolicy
    22 ClusterAutoscalerPolicy:
    23 Type: AWS::IAM::ManagedPolicy
    24 Properties:
    25 ManagedPolicyName: {{ AUTOSCALER_EKS_IAM_POLICY }}
    26 PolicyDocument:
    27 Version: '2012-10-17'
    28 Statement:
    29 - Effect: Allow
    30 Action:
    31 - autoscaling:DescribeAutoScalingGroups
    32 - autoscaling:DescribeAutoScalingInstances
    33 - autoscaling:DescribeLaunchConfigurations
    34 - autoscaling:DescribeTags
    35 - autoscaling:SetDesiredCapacity
    36 - autoscaling:TerminateInstanceInAutoScalingGroup
    37 - ec2:DescribeLaunchTemplateVersions
    38 Resource:
    39 - "*"
  10. Save your state:

    root@rok-tools:~/ops/deployments# rok-j2 deploy/env.autoscaler-eks-iam.j2 \ > -o deploy/env.autoscaler-eks-iam
  11. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit \ > -am "Create IAM Role for Cluster Autoscaler"
  12. Deploy the CloudFormation stack:

    root@rok-tools:~/ops/deployments# aws cloudformation deploy \ > --stack-name ${AUTOSCALER_EKS_IAM_CF?} \ > --template-file rok/eks/autoscaler-eks-iam-resources.yaml \ > --capabilities CAPABILITY_NAMED_IAM Waiting for changeset to be created.. Waiting for stack create/update to complete Successfully created/updated stack - cluster-autoscaler-us-west-2-arrikto-cluster

    Troubleshooting

    AccessDenied

    If the above command fails with an error message similar to the following:

    An error occurred (AccessDenied) when calling the DescribeStacks operation: User: arn:aws:iam::123456789012:user/user is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-east-1:123456789012:stack/rok-us-east-1-arriko-cluster-cluster-autoscaler/e84c63f0-3247-11ec-9c73-0a316e131472

    it means that your IAM user does not have sufficient permissions to perform an action necessary to deploy an AWS CloudFormation stack.

    To proceed, Check Your Environment and contact your AWS administrator to grant sufficient permissions to your IAM user or deploy the AWS CloudFormation stack for you.

    Failed to create/update the stack

    If the above command fails with an error message similar to the following:

    Failed to create/update the stack. Run the following command to fetch the list of events leading up to the failure aws cloudformation describe-stack-events --stack-name rok-us-east-1-arriko-cluster-cluster-autoscaler

    describe the events of the CloudFormation stack to identify the root cause of the failure:

    root@rok-tools:~/ops/deployments# aws cloudformation describe-stack-events --stack-name ${AUTOSCALER_EKS_IAM_CF?}
    • A stack event like the following:

      { "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/rok-us-east-1-arriko-cluster-cluster-autoscaler/599bc930-7b3f-11eb-ac1c-029efe3a90a0", "EventId": "rok-us-east-1-arriko-cluster-cluster-autoscaler-CREATE_FAILED-2021-03-02T10:09:27.457Z", "StackName": "rok-us-east-1-arriko-cluster-cluster-autoscaler", "LogicalResourceId": "ClusterAutoscalerRole", "PhysicalResourceId": "", "ResourceType": "AWS::IAM::Role", "Timestamp": "2021-03-02T10:09:27.457000+00:00", "ResourceStatus": "CREATE_FAILED", "ResourceStatusReason": "rok-us-east-1-arrikto-cluster-cluster-autoscaler already exists in stack arn:aws:cloudformation:es-east-1:123456789012:stack/rok-us-east-1-arrikto-another-cluster-cluster-autoscaler/e84c63f0-3247-11ec-9c73-0a316e131472", "ResourceProperties": "{\"ManagedPolicyArns\":[\"arn:aws:iam::123456789012:policy/rok-us-east-1-arrikto-cluster-cluster-autoscaler\"],\"RoleName\":\"rok-us-east-1-arrikto-cluster-cluster-autoscaler\",\"AssumeRolePolicyDocument\":{\"Version\":\"2012-10-17\",\"Statement\":[{\"Condition\":{\"StringEquals\":{\"oidc.eks.eu-central-1.amazonaws.com/id/123456789ABCDEFGHIJKLMNOPQRSTUVW:sub\":\"system:serviceaccount:kube-system:cluster-autoscaler\"}},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws:iam::123456789012:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/123456789ABCDEFGHIJKLMNOPQRSTUVW\"}}]}}" }

      means that the IAM role or IAM policy that the AWS CloudFormation stack defines already exist, leading to name conflicts.

      To proceed, go back to step 3, specify a different name for the resources that already exist and follow the rest of the guide.

    • A stack event like the following:

      { "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/rok-us-east-1-arriko-cluster-cluster-autoscaler/415eef80-7b46-11eb-b047-06980f530fec", "EventId": "rok-us-east-1-arriko-cluster-cluster-autoscaler-CREATE_FAILED-2021-03-02T10:09:27.457Z", "StackName": "rok-us-east-1-arriko-cluster-cluster-autoscaler", "LogicalResourceId": "ClusterAutoscalerRole", "PhysicalResourceId": "", "ResourceType": "AWS::IAM::Role", "Timestamp": "2021-03-02T10:58:54.216000+00:00", "ResourceStatus": "CREATE_FAILED", "ResourceStatusReason": "API: iam:CreateRole User: arn:aws:iam::123456789012:user/user is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::123456789012:role/rok-us-east-1-arrikto-cluster-cluster-autoscaler", "ResourceProperties": "{\"ManagedPolicyArns\":[\"arn:aws:iam::123456789012:policy/rok-us-east-1-arrikto-cluster-cluster-autoscaler\"],\"RoleName\":\"rok-us-east-1-arrikto-cluster-cluster-autoscaler\",\"AssumeRolePolicyDocument\":{\"Version\":\"2012-10-17\",\"Statement\":[{\"Condition\":{\"StringEquals\":{\"oidc.eks.eu-central-1.amazonaws.com/id/123456789ABCDEFGHIJKLMNOPQRSTUVW:sub\":\"system:serviceaccount:kube-system:cluster-autoscaler\"}},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws:iam::123456789012:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/123456789ABCDEFGHIJKLMNOPQRSTUVW\"}}]}}" }

      means that your IAM user does not have sufficient permissions to create the resources that the AWS CloudFormation stack defines.

      To proceed, Check Your Environment and contact your AWS administrator to grant your IAM user sufficient permissions or deploy the AWS CloudFormation stack for you.

    ValidationError

    If the above command fails with an error message similar to the following:

    An error occurred (ValidationError) when calling the CreateChangeSet operation: Stack:arn:aws:cloudformation:us-east-1:123456789012:stack/rok-us-east-1-arriko-cluster-cluster-autoscaler/671606f0-eb2b-11eb-8afb-0217413c9ed2 is in ROLLBACK_COMPLETE state and can not be updated.

    delete the stack and deploy it again.

  13. Mark your progress:

    root@rok-tools:~/ops/deployments# export DATE=$(date -u "+%Y-%m-%dT%H.%M.%SZ")
    root@rok-tools:~/ops/deployments# git tag \ > -a deploy/${DATE?}/develop/autoscaler-eks-iam \ > -m "Create IAM Role for Cluster Autoscaler"

Verify

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Restore the required context from previous sections:

    root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,autoscaler-eks-iam})
    root@rok-tools:~/ops/deployments# export AWS_ACCOUNT_ID AUTOSCALER_EKS_IAM_ROLE
  3. Verify that the IAM role exists:

    root@rok-tools:~/ops/deployments# aws iam get-role \ > --role-name ${AUTOSCALER_EKS_IAM_ROLE?} \ > --query Role.RoleName \ > --output text && echo OK cluster-autoscaler-arrikto-cluster OK
  4. Verify that the IAM role provides full access to all the necessary resources, that is, the output of the following command is allowed:

    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/autoscaler-eks-iam.yaml.j2)" \ > | jq -r '.EvaluationResults[].EvalDecision' \ > | sort -u allowed

    Troubleshooting

    Output contains ‘implicitDeny’ or ‘explicitDeny’

    If the output of simulate-principal-policy contains implicitDeny or explicitDeny, this means that the IAM role does not have sufficient permissions to provide full access to all the necessary resources. To proceed, create an IAM role following the Procedure.

Summary

You have successfully created an IAM role for the Cluster Autoscaler on your EKS cluster.

What’s Next

The next step is to deploy the Cluster Autoscaler on your EKS cluster.