Create IAM Role for AWS Load Balancer Controller

In this section you will create an IAM role for the service account that AWS Load Balancer Controller will run with so that it has permissions to manage AWS resources for your Application Load Balancer.

Fast Forward

If you have exposed EKF with ALB, it means you have already created an IAM role for AWS Load Balancer Controller. In this case, you do not need to create a new IAM role, because you are going to use the same ALB to expose Serving Models. Proceed to the What’s Next section.

Choose one of the following options to create an IAM role for AWS Load Balancer Controller:

Check Your Environment

To create this role, you are going to deploy a CloudFormation stack. When working with AWS CloudFormation stacks to manage resources, not only do you need sufficient permissions on AWS CloudFormation, but also on the underlying resources that are defined in the template.

In order to create an IAM role with proper IAM policies attached to it for the AWS Load Balancer Controller using AWS CloudFormation you need permissions for the following actions:

  • Deploy AWS CloudFormation stacks.
  • Create IAM policies.
  • Create IAM roles.

Note

If you do not have the above permissions, contact your AWS administrator to grant sufficient permissions to your IAM user or deploy the below AWS CloudFormation stack for you.

Option 1: Create IAM Role for AWS Load Balancer Controller Automatically (preferred)

Under construction

This section is a work in progress. Use Option 2: Create IAM Role for AWS Load Balancer Controller Manually instead.

Create an IAM role for AWS Load Balancer Controller by following the on-screen instructions on the rok-deploy user interface.

If rok-deploy is not already running, start it with:

root@rok-tools:~# rok-deploy --run-from serving-eks-alb-iam-alb
../../../../_images/eks-alb-iam-alb1.png

Proceed to the Summary section.

Option 2: Create IAM Role for AWS Load Balancer Controller Manually

If you want to create an IAM role for AWS Load Balancer Controller manually, follow the instructions below.

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Restore the required context from previous sections:

    root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,eks-cluster,\ > eks-identity})
    root@rok-tools:~/ops/deployments# export AWS_ACCOUNT_ID AWS_DEFAULT_REGION \ > EKS_CLUSTER EKS_CLUSTER_OIDC
  3. Specify the IAM policy name for AWS Load Balancer Controller:

    root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_POLICY=rok\ > -${AWS_DEFAULT_REGION?}-${EKS_CLUSTER?}-alb
  4. Verify that the IAM policy name you specified is not longer than 128 characters:

    root@rok-tools:~/ops/deployments# [[ ${#EKS_ALB_IAM_ALB_POLICY} -le 128 ]] && echo OK || echo FAIL OK

    Troubleshooting

    The output of the command is FAIL

    Go back to step 3 and specify a shorter name. Ensure the new names is not already in use.

  5. Specify the IAM role name and description for AWS Load Balancer Controller:

    root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_ROLE=rok-\ > ${AWS_DEFAULT_REGION?}-${EKS_CLUSTER?}-alb
    root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_ROLE_DESC="AWS Load Balancer Controller"
  6. Verify that the IAM role name you specified is not longer than 64 characters:

    root@rok-tools:~/ops/deployments# [[ ${#EKS_ALB_IAM_ALB_ROLE} -le 64 ]] && echo OK || echo FAIL OK

    Troubleshooting

    The output of the command is FAIL

    Go back to step 5 and specify a shorter name. Ensure the new name is not already in use.

  7. Specify the service account name and namespace that AWS Load Balancer Controller will run with:

    root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_SA_NAMESPACE=kube-system
    root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_SA_NAME=aws-load-balancer-controller
  8. Set the name of the CloudFormation stack you will deploy:

    root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_CF=rok-${AWS_DEFAULT_REGION?}\ > -${EKS_CLUSTER?}-alb
  9. Verify that the CloudFormation stack name you specified is not longer than 128 characters:

    root@rok-tools:~/ops/deployments# [[ ${#EKS_ALB_IAM_ALB_CF} -le 128 ]] && echo OK || echo FAIL OK

    Troubleshooting

    The output of the command is FAIL

    Go back to step 8 and specify a shorter name. Ensure the new name is not already in use.

  10. Generate the AWS CloudFormation stack:

    root@rok-tools:~/ops/deployments# j2 rok/eks/eks-alb-alb-resources.yaml.j2 \ > -o rok/eks/eks-alb-alb-resources.yaml

    Alternatively, download the eks-alb-alb-resources CloudFormation template provided below and use it locally.

    eks-alb-alb-resources.yaml.j2
    1AWSTemplateFormatVersion: "2010-09-09"
    2
    3Description: Amazon EKS - AWS Load Balancer Controller IAM Role and Policy
    4-181
    4
    5Metadata:
    6 Rok::StackName: {{EKS_ALB_IAM_ALB_CF}}
    7
    8Resources:
    9 RokALBRole:
    10 Type: AWS::IAM::Role
    11 Properties:
    12 RoleName: {{EKS_ALB_IAM_ALB_ROLE}}
    13 AssumeRolePolicyDocument:
    14 Version: '2012-10-17'
    15 Statement:
    16 - Effect: Allow
    17 Action: sts:AssumeRoleWithWebIdentity
    18 Principal:
    19 Federated: arn:aws:iam::{{AWS_ACCOUNT_ID}}:oidc-provider/{{EKS_CLUSTER_OIDC}}
    20 Condition:
    21 StringEquals:
    22 {{EKS_CLUSTER_OIDC}}:sub: system:serviceaccount:{{EKS_ALB_IAM_ALB_SA_NAMESPACE}}:{{EKS_ALB_IAM_ALB_SA_NAME}}
    23 Description: {{EKS_ALB_IAM_ALB_ROLE_DESC}}
    24 ManagedPolicyArns:
    25 - Ref: RokALBPolicy
    26 RokALBPolicy:
    27 Type: AWS::IAM::ManagedPolicy
    28 Properties:
    29 ManagedPolicyName: {{EKS_ALB_IAM_ALB_POLICY}}
    30 PolicyDocument:
    31 Version: '2012-10-17'
    32 Statement:
    33 - Effect: Allow
    34 Action:
    35 - iam:CreateServiceLinkedRole
    36 Resource: "*"
    37 Condition:
    38 StringEquals:
    39 iam:AWSServiceName: elasticloadbalancing.amazonaws.com
    40 - Effect: Allow
    41 Action:
    42 - ec2:DescribeAccountAttributes
    43 - ec2:DescribeAddresses
    44 - ec2:DescribeAvailabilityZones
    45 - ec2:DescribeInternetGateways
    46 - ec2:DescribeVpcs
    47 - ec2:DescribeVpcPeeringConnections
    48 - ec2:DescribeSubnets
    49 - ec2:DescribeSecurityGroups
    50 - ec2:DescribeInstances
    51 - ec2:DescribeNetworkInterfaces
    52 - ec2:DescribeTags
    53 - ec2:GetCoipPoolUsage
    54 - ec2:DescribeCoipPools
    55 - elasticloadbalancing:DescribeLoadBalancers
    56 - elasticloadbalancing:DescribeLoadBalancerAttributes
    57 - elasticloadbalancing:DescribeListeners
    58 - elasticloadbalancing:DescribeListenerCertificates
    59 - elasticloadbalancing:DescribeSSLPolicies
    60 - elasticloadbalancing:DescribeRules
    61 - elasticloadbalancing:DescribeTargetGroups
    62 - elasticloadbalancing:DescribeTargetGroupAttributes
    63 - elasticloadbalancing:DescribeTargetHealth
    64 - elasticloadbalancing:DescribeTags
    65 Resource: "*"
    66 - Effect: Allow
    67 Action:
    68 - cognito-idp:DescribeUserPoolClient
    69 - acm:ListCertificates
    70 - acm:DescribeCertificate
    71 - iam:ListServerCertificates
    72 - iam:GetServerCertificate
    73 - waf-regional:GetWebACL
    74 - waf-regional:GetWebACLForResource
    75 - waf-regional:AssociateWebACL
    76 - waf-regional:DisassociateWebACL
    77 - wafv2:GetWebACL
    78 - wafv2:GetWebACLForResource
    79 - wafv2:AssociateWebACL
    80 - wafv2:DisassociateWebACL
    81 - shield:GetSubscriptionState
    82 - shield:DescribeProtection
    83 - shield:CreateProtection
    84 - shield:DeleteProtection
    85 Resource: "*"
    86 - Effect: Allow
    87 Action:
    88 - ec2:AuthorizeSecurityGroupIngress
    89 - ec2:RevokeSecurityGroupIngress
    90 Resource: "*"
    91 - Effect: Allow
    92 Action:
    93 - ec2:CreateSecurityGroup
    94 Resource: "*"
    95 - Effect: Allow
    96 Action:
    97 - ec2:CreateTags
    98 Resource: arn:aws:ec2:*:*:security-group/*
    99 Condition:
    100 StringEquals:
    101 ec2:CreateAction: CreateSecurityGroup
    102 'Null':
    103 aws:RequestTag/elbv2.k8s.aws/cluster: 'false'
    104 - Effect: Allow
    105 Action:
    106 - ec2:CreateTags
    107 - ec2:DeleteTags
    108 Resource: arn:aws:ec2:*:*:security-group/*
    109 Condition:
    110 'Null':
    111 aws:RequestTag/elbv2.k8s.aws/cluster: 'true'
    112 aws:ResourceTag/elbv2.k8s.aws/cluster: 'false'
    113 - Effect: Allow
    114 Action:
    115 - ec2:AuthorizeSecurityGroupIngress
    116 - ec2:RevokeSecurityGroupIngress
    117 - ec2:DeleteSecurityGroup
    118 Resource: "*"
    119 Condition:
    120 'Null':
    121 aws:ResourceTag/elbv2.k8s.aws/cluster: 'false'
    122 - Effect: Allow
    123 Action:
    124 - elasticloadbalancing:CreateLoadBalancer
    125 - elasticloadbalancing:CreateTargetGroup
    126 Resource: "*"
    127 Condition:
    128 'Null':
    129 aws:RequestTag/elbv2.k8s.aws/cluster: 'false'
    130 - Effect: Allow
    131 Action:
    132 - elasticloadbalancing:CreateListener
    133 - elasticloadbalancing:DeleteListener
    134 - elasticloadbalancing:CreateRule
    135 - elasticloadbalancing:DeleteRule
    136 Resource: "*"
    137 - Effect: Allow
    138 Action:
    139 - elasticloadbalancing:AddTags
    140 - elasticloadbalancing:RemoveTags
    141 Resource:
    142 - arn:aws:elasticloadbalancing:*:*:targetgroup/*/*
    143 - arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*
    144 - arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*
    145 Condition:
    146 'Null':
    147 aws:RequestTag/elbv2.k8s.aws/cluster: 'true'
    148 aws:ResourceTag/elbv2.k8s.aws/cluster: 'false'
    149 - Effect: Allow
    150 Action:
    151 - elasticloadbalancing:AddTags
    152 - elasticloadbalancing:RemoveTags
    153 Resource:
    154 - arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*
    155 - arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*
    156 - arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*
    157 - arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*
    158 - Effect: Allow
    159 Action:
    160 - elasticloadbalancing:ModifyLoadBalancerAttributes
    161 - elasticloadbalancing:SetIpAddressType
    162 - elasticloadbalancing:SetSecurityGroups
    163 - elasticloadbalancing:SetSubnets
    164 - elasticloadbalancing:DeleteLoadBalancer
    165 - elasticloadbalancing:ModifyTargetGroup
    166 - elasticloadbalancing:ModifyTargetGroupAttributes
    167 - elasticloadbalancing:DeleteTargetGroup
    168 Resource: "*"
    169 Condition:
    170 'Null':
    171 aws:ResourceTag/elbv2.k8s.aws/cluster: 'false'
    172 - Effect: Allow
    173 Action:
    174 - elasticloadbalancing:RegisterTargets
    175 - elasticloadbalancing:DeregisterTargets
    176 Resource: arn:aws:elasticloadbalancing:*:*:targetgroup/*/*
    177 - Effect: Allow
    178 Action:
    179 - elasticloadbalancing:SetWebAcl
    180 - elasticloadbalancing:ModifyListener
    181 - elasticloadbalancing:AddListenerCertificates
    182 - elasticloadbalancing:RemoveListenerCertificates
    183 - elasticloadbalancing:ModifyRule
    184 Resource: "*"
  11. Save your state:

    root@rok-tools:~/ops/deployments# j2 deploy/env.eks-alb-iam-alb.j2 \ > -o deploy/env.eks-alb-iam-alb
  12. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit -am "Create IAM Role for AWS Load Balancer Controller"
  13. Mark your progress:

    root@rok-tools:~/ops/deployments# export DATE=$(date -u "+%Y-%m-%dT%H.%M.%SZ")
    root@rok-tools:~/ops/deployments# git tag \ > -a deploy/${DATE?}/release-2.0/eks-alb-iam-alb \ > -m "Create IAM Role for AWS Load Balancer Controller"
  14. Deploy the CloudFormation stack:

    root@rok-tools:~/ops/deployments# aws cloudformation deploy \ > --stack-name ${EKS_ALB_IAM_ALB_CF?} \ > --template-file rok/eks/eks-alb-alb-resources.yaml \ > --capabilities CAPABILITY_NAMED_IAM Waiting for changeset to be created.. Waiting for stack create/update to complete Successfully created/updated stack - rok-us-west-2-arrikto-cluster-alb

    Troubleshooting

    AccessDenied

    If the above command fails with an error message similar to the following:

    An error occurred (AccessDenied) when calling the DescribeStacks operation: User: arn:aws:iam::123456789012:user/user is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-west-2:123456789012:stack/rok-us-west-2-arrikto-cluster-alb

    it means that your IAM user does not have sufficient permissions to perform an action necessary to deploy an AWS CloudFormation stack.

    To proceed, Check Your Environment and contact your AWS administrator to grant sufficient permissions to your IAM user or deploy the AWS CloudFormation stack for you.

    Failed to create/update the stack

    If the above command fails with an error message similar to the following:

    Failed to create/update the stack. Run the following command to fetch the list of events leading up to the failure aws cloudformation describe-stack-events --stack-name rok-us-west-2-arrikto-cluster-alb

    describe the events of the CloudFormation stack to identify the root cause of the failure:

    root@rok-tools:~/ops/deployments# aws cloudformation describe-stack-events --stack-name ${EKS_ALB_IAM_ALB_CF?}
    • A stack event like the following:

      { "StackId": "arn:aws:cloudformation:us-west-2:123456789012:stack/rok-us-west-2-arrikto-cluster-alb/599bc930-7b3f-11eb-ac1c-029efe3a90a0", "EventId": "RokALBRole-CREATE_FAILED-2021-03-02T10:09:27.457Z", "StackName": "rok-us-west-2-arrikto-cluster-alb", "LogicalResourceId": "RokALBRole", "PhysicalResourceId": "", "ResourceType": "AWS::IAM::Role", "Timestamp": "2021-03-02T10:09:27.457000+00:00", "ResourceStatus": "CREATE_FAILED", "ResourceStatusReason": "rok-us-west-2-arrikto-cluster-alb already exists", "ResourceProperties": "{\"ManagedPolicyArns\":[\"arn:aws:iam::123456789012:policy/rok-us-west-2-arrikto-cluster-alb\"],\"RoleName\":\"rok-us-west-2-arrikto-cluster-alb\",\"Description\":\"AWS Load Balancer Controller\",\"AssumeRolePolicyDocument\":{\"Version\":\"2012-10-17\",\"Statement\":[{\"Condition\":{\"StringEquals\":{\"oidc.eks.us-west-2.amazonaws.com/id/352A7999E682D224D5A47B738D375237:sub\":\"system:serviceaccount:kube-system:aws-load-balancer-controller\"}},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/352A7999E682D224D5A47B738D375237\"}}]}}" }

      means that the IAM Role or the IAM Policy that the AWS CloudFormation stack defines already exist, leading to name conflicts.

      To proceed, go back to step 3, specify a different name for resources that already exist and follow the rest of the guide.

    • A stack event like the following:

      { "StackId": "arn:aws:cloudformation:us-west-2:123456789012:stack/rok-us-west-2-arrikto-cluster-alb/415eef80-7b46-11eb-b047-06980f530fec", "EventId": "RokALBRole-CREATE_FAILED-2021-03-02T10:58:54.216Z", "StackName": "rok-us-west-2-arrikto-cluster-alb", "LogicalResourceId": "RokALBRole", "PhysicalResourceId": "", "ResourceType": "AWS::IAM::Role", "Timestamp": "2021-03-02T10:58:54.216000+00:00", "ResourceStatus": "CREATE_FAILED", "ResourceStatusReason": "API: iam:CreateRole User: arn:aws:iam::123456789012:user/user is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::123456789012:role/rok-us-west-2-arrikto-cluster-alb", "ResourceProperties": "{\"ManagedPolicyArns\":[\"arn:aws:iam::123456789012:policy/rok-us-west-2-arrikto-cluster-alb\"],\"RoleName\":\"rok-us-west-2-arrikto-cluster-alb\",\"Description\":\"AWS Load Balancer Controller\",\"AssumeRolePolicyDocument\":{\"Version\":\"2012-10-17\",\"Statement\":[{\"Condition\":{\"StringEquals\":{\"oidc.eks.us-west-2.amazonaws.com/id/352A7999E682D224D5A47B738D375237:sub\":\"system:serviceaccount:kube-system:aws-load-balancer-controller\"}},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/352A7999E682D224D5A47B738D375237\"}}]}}" }

      means that your IAM user does not have sufficient permissions to create the resources that the AWS CloudFormation stack defines.

      To proceed, Check Your Environment and contact your AWS administrator to grant your IAM user sufficient permissions or deploy the AWS CloudFormation stack for you.

    ValidationError

    If the above command fails with an error message similar to the following:

    An error occurred (ValidationError) when calling the CreateChangeSet operation: Stack:arn:aws:cloudformation:us-west-2:123456789012:stack/rok-us-west-2-arrikto-cluster-alb/671606f0-eb2b-11eb-8afb-0217413c9ed2 is in ROLLBACK_COMPLETE state and can not be updated.

    delete the stack and deploy it again.

Upgrade from ALB Ingress Controller to AWS Load Balancer Controller

Note

In case you are upgrading from ALB Ingress Controller to AWS Load Balancer Controller, you have to assign some extra permissions to the IAM role so that the new controller will be able to manage existing AWS resources. To do so, follow the extra steps below.

  1. Create the extra policy to allow the AWS Load Balancer Controller pod to manage existing AWS resources created previously by ALB Ingress Controller:

    root@rok-tools:~/ops/deployments# aws iam create-policy \ > --policy-name AWSLoadBalancerControllerExtraIAMPolicy \ > --policy-document file://rok/aws-load-balancer-controller/iam-policy-albv2-extra.json

    Alternatively, save the JSON policy document provided below or download iam-policy-albv2-extra.json and use it locally.

  2. Attach the policy to the previously created IAM role:

    root@rok-tools:~/ops/deployments# aws iam attach-role-policy \ > --role-name ${EKS_ALB_IAM_ALB_ROLE?} \ > --policy-arn=arn:aws:iam::${AWS_ACCOUNT_ID?}:policy/AWSLoadBalancerControllerExtraIAMPolicy

Verify

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Restore the required context from previous sections:

    root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,eks-alb-iam-alb})
    root@rok-tools:~/ops/deployments# export AWS_ACCOUNT_ID EKS_ALB_IAM_ALB_ROLE
  3. Verify that the IAM role exists:

    root@rok-tools:~/ops/deployments# aws iam get-role \ > --role-name ${EKS_ALB_IAM_ALB_ROLE?} \ > --query Role.RoleName \ > --output text && echo OK rok-us-west-2-arrikto-cluster-alb OK
  4. Verify that the role has the necessary permissions to manage AWS resources for your Application Load Balancer:

    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-all.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed
    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-ec2-createtags.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed
    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-ec2-createdeletetags.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed
    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-delete-secgroup.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed
    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-elb-create.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed
    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-elb-addremovetags.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed
    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-elb-modifytargetgroup.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed
    root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-elb-registertargets.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed

Summary

You have successfully created the IAM role for AWS Load Balancer Controller.

What’s Next

The next step is to deploy AWS Load Balancer Controller.