Create IAM Role for AWS Load Balancer Controller¶
In this section you will create an IAM role for the service account that AWS Load Balancer Controller will run with so that it has permissions to manage AWS resources for your Application Load Balancer.
Fast Forward
If you have exposed EKF with ALB, it means you have already created an IAM role for AWS Load Balancer Controller. In this case, you do not need to create a new IAM role, because you are going to use the same ALB to expose KFServing Models. Proceed to the What’s Next section.
Fast Forward
If you already have an IAM role for AWS Load Balancer Controller, expand this box to fast-forward.
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:~# cd ~/ops/deploymentsRestore the required context from previous sections:
root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,eks-cluster})root@rok-tools:~/ops/deployments# export AWS_DEFAULT_REGION EKS_CLUSTERSpecify the name of the IAM role for AWS Load Balancer Controller:
root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_ROLE=<ROLE>Replace
<ROLE>
with the name of the IAM role for AWS Load Balancer Controller, for example:root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_ROLE=rok-\ > ${AWS_DEFAULT_REGION?}-${EKS_CLUSTER?}-albSave your state:
root@rok-tools:~/ops/deployments# j2 deploy/env.eks-alb-iam-alb.j2 \ > -o deploy/env.eks-alb-iam-albCommit your changes:
root@rok-tools:~/ops/deployments# git commit -am "Create IAM Role for AWS Load Balancer Controller"Proceed to the Verify section.
See also
Choose one of the following options to create an IAM role for AWS Load Balancer Controller:
- Option 1: Create IAM Role for AWS Load Balancer Controller Automatically (preferred).
- Option 2: Create IAM Role for AWS Load Balancer Controller Manually.
Overview
What You’ll Need¶
- A configured management environment.
- Your clone of the Arrikto GitOps repository.
- An existing EKS cluster.
Check Your Environment¶
To create this role, you are going to deploy a CloudFormation stack. When working with AWS CloudFormation stacks to manage resources, not only do you need sufficient permissions on AWS CloudFormation, but also on the underlying resources that are defined in the template.
In order to create an IAM role with proper IAM policies attached to it for the AWS Load Balancer Controller using AWS CloudFormation you need permissions for the following actions:
- Deploy AWS CloudFormation stacks.
- Create IAM policies.
- Create IAM roles.
Note
If you do not have the above permissions, contact your AWS administrator to grant sufficient permissions to your IAM user or deploy the below AWS CloudFormation stack for you.
Option 1: Create IAM Role for AWS Load Balancer Controller Automatically (preferred)¶
Under construction
This section is a work in progress. Use Option 2: Create IAM Role for AWS Load Balancer Controller Manually instead.
Create an IAM role for AWS Load Balancer Controller by following the on-screen
instructions on the rok-deploy
user interface.
If rok-deploy
is not already running, start it with:

Proceed to the Summary section.
Option 2: Create IAM Role for AWS Load Balancer Controller Manually¶
If you want to create an IAM role for AWS Load Balancer Controller manually, follow the instructions below.
Procedure¶
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:~# cd ~/ops/deploymentsRestore the required context from previous sections:
root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,eks-cluster,\ > eks-identity})root@rok-tools:~/ops/deployments# export AWS_ACCOUNT_ID AWS_DEFAULT_REGION \ > EKS_CLUSTER EKS_CLUSTER_OIDCSpecify the IAM policy name for AWS Load Balancer Controller:
root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_POLICY=rok\ > -${AWS_DEFAULT_REGION?}-${EKS_CLUSTER?}-albVerify that the IAM policy name you specified is not longer than 128 characters:
root@rok-tools:~/ops/deployments# [[ ${#EKS_ALB_IAM_ALB_POLICY} -le 128 ]] && echo OK || echo FAIL OKTroubleshooting
The output of the command is FAIL
Go back to step 3 and specify a shorter name. Ensure the new names is not already in use.
Specify the IAM role name and description for AWS Load Balancer Controller:
root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_ROLE=rok-\ > ${AWS_DEFAULT_REGION?}-${EKS_CLUSTER?}-albroot@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_ROLE_DESC="AWS Load Balancer Controller"Verify that the IAM role name you specified is not longer than 64 characters:
root@rok-tools:~/ops/deployments# [[ ${#EKS_ALB_IAM_ALB_ROLE} -le 64 ]] && echo OK || echo FAIL OKTroubleshooting
The output of the command is FAIL
Go back to step 5 and specify a shorter name. Ensure the new name is not already in use.
Specify the service account name and namespace that AWS Load Balancer Controller will run with:
root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_SA_NAMESPACE=kube-systemroot@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_SA_NAME=aws-load-balancer-controllerSet the name of the CloudFormation stack you will deploy:
root@rok-tools:~/ops/deployments# export EKS_ALB_IAM_ALB_CF=rok-${AWS_DEFAULT_REGION?}\ > -${EKS_CLUSTER?}-albVerify that the CloudFormation stack name you specified is not longer than 128 characters:
root@rok-tools:~/ops/deployments# [[ ${#EKS_ALB_IAM_ALB_CF} -le 128 ]] && echo OK || echo FAIL OKTroubleshooting
The output of the command is FAIL
Go back to step 8 and specify a shorter name. Ensure the new name is not already in use.
Generate the AWS CloudFormation stack:
root@rok-tools:~/ops/deployments# j2 rok/eks/eks-alb-alb-resources.yaml.j2 \ > -o rok/eks/eks-alb-alb-resources.yamlAlternatively, download the
eks-alb-alb-resources
CloudFormation template provided below and use it locally.eks-alb-alb-resources.yaml.j21 AWSTemplateFormatVersion: "2010-09-09" 2 3 Description: Amazon EKS - AWS Load Balancer Controller IAM Role and Policy 4-162 4 5 Metadata: 6 Rok::StackName: {{EKS_ALB_IAM_ALB_CF}} 7 8 Resources: 9 RokALBRole: 10 Type: AWS::IAM::Role 11 Properties: 12 RoleName: {{EKS_ALB_IAM_ALB_ROLE}} 13 AssumeRolePolicyDocument: 14 Version: '2012-10-17' 15 Statement: 16 - Effect: Allow 17 Action: sts:AssumeRoleWithWebIdentity 18 Principal: 19 Federated: arn:aws:iam::{{AWS_ACCOUNT_ID}}:oidc-provider/{{EKS_CLUSTER_OIDC}} 20 Condition: 21 StringEquals: 22 {{EKS_CLUSTER_OIDC}}:sub: system:serviceaccount:{{EKS_ALB_IAM_ALB_SA_NAMESPACE}}:{{EKS_ALB_IAM_ALB_SA_NAME}} 23 Description: {{EKS_ALB_IAM_ALB_ROLE_DESC}} 24 ManagedPolicyArns: 25 - Ref: RokALBPolicy 26 RokALBPolicy: 27 Type: AWS::IAM::ManagedPolicy 28 Properties: 29 ManagedPolicyName: {{EKS_ALB_IAM_ALB_POLICY}} 30 PolicyDocument: 31 Version: '2012-10-17' 32 Statement: 33 - Effect: Allow 34 Action: 35 - iam:CreateServiceLinkedRole 36 - ec2:DescribeAccountAttributes 37 - ec2:DescribeAddresses 38 - ec2:DescribeInternetGateways 39 - ec2:DescribeVpcs 40 - ec2:DescribeSubnets 41 - ec2:DescribeSecurityGroups 42 - ec2:DescribeInstances 43 - ec2:DescribeNetworkInterfaces 44 - ec2:DescribeTags 45 - elasticloadbalancing:DescribeLoadBalancers 46 - elasticloadbalancing:DescribeLoadBalancerAttributes 47 - elasticloadbalancing:DescribeListeners 48 - elasticloadbalancing:DescribeListenerCertificates 49 - elasticloadbalancing:DescribeSSLPolicies 50 - elasticloadbalancing:DescribeRules 51 - elasticloadbalancing:DescribeTargetGroups 52 - elasticloadbalancing:DescribeTargetGroupAttributes 53 - elasticloadbalancing:DescribeTargetHealth 54 - elasticloadbalancing:DescribeTags 55 Resource: "*" 56 - Effect: Allow 57 Action: 58 - cognito-idp:DescribeUserPoolClient 59 - acm:ListCertificates 60 - acm:DescribeCertificate 61 - iam:ListServerCertificates 62 - iam:GetServerCertificate 63 - waf-regional:GetWebACL 64 - waf-regional:GetWebACLForResource 65 - waf-regional:AssociateWebACL 66 - waf-regional:DisassociateWebACL 67 - wafv2:GetWebACL 68 - wafv2:GetWebACLForResource 69 - wafv2:AssociateWebACL 70 - wafv2:DisassociateWebACL 71 - shield:GetSubscriptionState 72 - shield:DescribeProtection 73 - shield:CreateProtection 74 - shield:DeleteProtection 75 Resource: "*" 76 - Effect: Allow 77 Action: 78 - ec2:AuthorizeSecurityGroupIngress 79 - ec2:RevokeSecurityGroupIngress 80 Resource: "*" 81 - Effect: Allow 82 Action: 83 - ec2:CreateSecurityGroup 84 Resource: "*" 85 - Effect: Allow 86 Action: 87 - ec2:CreateTags 88 Resource: arn:aws:ec2:*:*:security-group/* 89 Condition: 90 StringEquals: 91 ec2:CreateAction: CreateSecurityGroup 92 'Null': 93 aws:RequestTag/elbv2.k8s.aws/cluster: 'false' 94 - Effect: Allow 95 Action: 96 - ec2:CreateTags 97 - ec2:DeleteTags 98 Resource: arn:aws:ec2:*:*:security-group/* 99 Condition: 100 'Null': 101 aws:RequestTag/elbv2.k8s.aws/cluster: 'true' 102 aws:ResourceTag/elbv2.k8s.aws/cluster: 'false' 103 - Effect: Allow 104 Action: 105 - ec2:AuthorizeSecurityGroupIngress 106 - ec2:RevokeSecurityGroupIngress 107 - ec2:DeleteSecurityGroup 108 Resource: "*" 109 Condition: 110 'Null': 111 aws:ResourceTag/elbv2.k8s.aws/cluster: 'false' 112 - Effect: Allow 113 Action: 114 - elasticloadbalancing:CreateLoadBalancer 115 - elasticloadbalancing:CreateTargetGroup 116 Resource: "*" 117 Condition: 118 'Null': 119 aws:RequestTag/elbv2.k8s.aws/cluster: 'false' 120 - Effect: Allow 121 Action: 122 - elasticloadbalancing:CreateListener 123 - elasticloadbalancing:DeleteListener 124 - elasticloadbalancing:CreateRule 125 - elasticloadbalancing:DeleteRule 126 Resource: "*" 127 - Effect: Allow 128 Action: 129 - elasticloadbalancing:AddTags 130 - elasticloadbalancing:RemoveTags 131 Resource: 132 - arn:aws:elasticloadbalancing:*:*:targetgroup/*/* 133 - arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/* 134 - arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/* 135 Condition: 136 'Null': 137 aws:RequestTag/elbv2.k8s.aws/cluster: 'true' 138 aws:ResourceTag/elbv2.k8s.aws/cluster: 'false' 139 - Effect: Allow 140 Action: 141 - elasticloadbalancing:ModifyLoadBalancerAttributes 142 - elasticloadbalancing:SetIpAddressType 143 - elasticloadbalancing:SetSecurityGroups 144 - elasticloadbalancing:SetSubnets 145 - elasticloadbalancing:DeleteLoadBalancer 146 - elasticloadbalancing:ModifyTargetGroup 147 - elasticloadbalancing:ModifyTargetGroupAttributes 148 - elasticloadbalancing:DeleteTargetGroup 149 Resource: "*" 150 Condition: 151 'Null': 152 aws:ResourceTag/elbv2.k8s.aws/cluster: 'false' 153 - Effect: Allow 154 Action: 155 - elasticloadbalancing:RegisterTargets 156 - elasticloadbalancing:DeregisterTargets 157 Resource: arn:aws:elasticloadbalancing:*:*:targetgroup/*/* 158 - Effect: Allow 159 Action: 160 - elasticloadbalancing:SetWebAcl 161 - elasticloadbalancing:ModifyListener 162 - elasticloadbalancing:AddListenerCertificates 163 - elasticloadbalancing:RemoveListenerCertificates 164 - elasticloadbalancing:ModifyRule 165 Resource: "*" Save your state:
root@rok-tools:~/ops/deployments# j2 deploy/env.eks-alb-iam-alb.j2 \ > -o deploy/env.eks-alb-iam-albCommit your changes:
root@rok-tools:~/ops/deployments# git commit -am "Create IAM Role for AWS Load Balancer Controller"Mark your progress:
root@rok-tools:~/ops/deployments# export DATE=$(date -u "+%Y-%m-%dT%H.%M.%SZ")root@rok-tools:~/ops/deployments# git tag \ > -a deploy/${DATE?}/release-1.5/eks-alb-iam-alb \ > -m "Create IAM Role for AWS Load Balancer Controller"Deploy the CloudFormation stack:
root@rok-tools:~/ops/deployments# aws cloudformation deploy \ > --stack-name ${EKS_ALB_IAM_ALB_CF?} \ > --template-file rok/eks/eks-alb-alb-resources.yaml \ > --capabilities CAPABILITY_NAMED_IAM Waiting for changeset to be created.. Waiting for stack create/update to complete Successfully created/updated stack - rok-us-west-2-arrikto-cluster-albTroubleshooting
AccessDenied
If the above command fails with an error message similar to the following:
An error occurred (AccessDenied) when calling the DescribeStacks operation: User: arn:aws:iam::123456789012:user/user is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-west-2:123456789012:stack/rok-us-west-2-arrikto-cluster-albit means that your IAM user does not have sufficient permissions to perform an action necessary to deploy an AWS CloudFormation stack.
To proceed, Check Your Environment and contact your AWS administrator to grant sufficient permissions to your IAM user or deploy the AWS CloudFormation stack for you.
Failed to create/update the stack
If the above command fails with an error message similar to the following:
Failed to create/update the stack. Run the following command to fetch the list of events leading up to the failure aws cloudformation describe-stack-events --stack-name rok-us-west-2-arrikto-cluster-albdescribe the events of the CloudFormation stack to identify the root cause of the failure:
root@rok-tools:~/ops/deployments# aws cloudformation describe-stack-events --stack-name ${EKS_ALB_IAM_ALB_CF?}A stack event like the following:
{ "StackId": "arn:aws:cloudformation:us-west-2:123456789012:stack/rok-us-west-2-arrikto-cluster-alb/599bc930-7b3f-11eb-ac1c-029efe3a90a0", "EventId": "RokALBRole-CREATE_FAILED-2021-03-02T10:09:27.457Z", "StackName": "rok-us-west-2-arrikto-cluster-alb", "LogicalResourceId": "RokALBRole", "PhysicalResourceId": "", "ResourceType": "AWS::IAM::Role", "Timestamp": "2021-03-02T10:09:27.457000+00:00", "ResourceStatus": "CREATE_FAILED", "ResourceStatusReason": "rok-us-west-2-arrikto-cluster-alb already exists", "ResourceProperties": "{\"ManagedPolicyArns\":[\"arn:aws:iam::123456789012:policy/rok-us-west-2-arrikto-cluster-alb\"],\"RoleName\":\"rok-us-west-2-arrikto-cluster-alb\",\"Description\":\"AWS Load Balancer Controller\",\"AssumeRolePolicyDocument\":{\"Version\":\"2012-10-17\",\"Statement\":[{\"Condition\":{\"StringEquals\":{\"oidc.eks.us-west-2.amazonaws.com/id/352A7999E682D224D5A47B738D375237:sub\":\"system:serviceaccount:kube-system:aws-load-balancer-controller\"}},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/352A7999E682D224D5A47B738D375237\"}}]}}" }means that the IAM Role or the IAM Policy that the AWS CloudFormation stack defines already exist, leading to name conflicts.
To proceed, go back to step 3, specify a different name for resources that already exist and follow the rest of the guide.
A stack event like the following:
{ "StackId": "arn:aws:cloudformation:us-west-2:123456789012:stack/rok-us-west-2-arrikto-cluster-alb/415eef80-7b46-11eb-b047-06980f530fec", "EventId": "RokALBRole-CREATE_FAILED-2021-03-02T10:58:54.216Z", "StackName": "rok-us-west-2-arrikto-cluster-alb", "LogicalResourceId": "RokALBRole", "PhysicalResourceId": "", "ResourceType": "AWS::IAM::Role", "Timestamp": "2021-03-02T10:58:54.216000+00:00", "ResourceStatus": "CREATE_FAILED", "ResourceStatusReason": "API: iam:CreateRole User: arn:aws:iam::123456789012:user/user is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::123456789012:role/rok-us-west-2-arrikto-cluster-alb", "ResourceProperties": "{\"ManagedPolicyArns\":[\"arn:aws:iam::123456789012:policy/rok-us-west-2-arrikto-cluster-alb\"],\"RoleName\":\"rok-us-west-2-arrikto-cluster-alb\",\"Description\":\"AWS Load Balancer Controller\",\"AssumeRolePolicyDocument\":{\"Version\":\"2012-10-17\",\"Statement\":[{\"Condition\":{\"StringEquals\":{\"oidc.eks.us-west-2.amazonaws.com/id/352A7999E682D224D5A47B738D375237:sub\":\"system:serviceaccount:kube-system:aws-load-balancer-controller\"}},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/352A7999E682D224D5A47B738D375237\"}}]}}" }means that your IAM user does not have sufficient permissions to create the resources that the AWS CloudFormation stack defines.
To proceed, Check Your Environment and contact your AWS administrator to grant your IAM user sufficient permissions or deploy the AWS CloudFormation stack for you.
ValidationError
If the above command fails with an error message similar to the following:
An error occurred (ValidationError) when calling the CreateChangeSet operation: Stack:arn:aws:cloudformation:us-west-2:123456789012:stack/rok-us-west-2-arrikto-cluster-alb/671606f0-eb2b-11eb-8afb-0217413c9ed2 is in ROLLBACK_COMPLETE state and can not be updated.delete the stack and deploy it again.
Note
In case you are upgrading from ALB Ingress Controller to AWS Load Balancer Controller, you have to assign some extra permissions to the IAM role so that the new controller will be able to manage existing AWS resources. To do so, follow the extra steps below.
Create the extra policy to allow the AWS Load Balancer Controller pod to manage existing AWS resources created previously by ALB Ingress Controller:
root@rok-tools:~/ops/deployments# aws iam create-policy \ > --policy-name AWSLoadBalancerControllerExtraIAMPolicy \ > --policy-document file://rok/aws-load-balancer-controller/iam-policy-albv2-extra.jsonAlternatively, save the JSON policy document provided below or download
iam-policy-albv2-extra.json
and use it locally.Attach the policy to the previously created IAM role:
root@rok-tools:~/ops/deployments# aws iam attach-role-policy \ > --role-name ${EKS_ALB_IAM_ALB_ROLE?} \ > --policy-arn=arn:aws:iam::${AWS_ACCOUNT_ID?}:policy/AWSLoadBalancerControllerExtraIAMPolicy
Verify¶
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:~# cd ~/ops/deploymentsRestore the required context from previous sections:
root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,eks-alb-iam-alb})root@rok-tools:~/ops/deployments# export AWS_ACCOUNT_ID EKS_ALB_IAM_ALB_ROLEVerify that the IAM role exists:
root@rok-tools:~/ops/deployments# aws iam get-role \ > --role-name ${EKS_ALB_IAM_ALB_ROLE?} \ > --query Role.RoleName \ > --output text && echo OK rok-us-west-2-arrikto-cluster-alb OKVerify that the role has the necessary permissions to manage AWS resources for your Application Load Balancer:
root@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-all.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowedroot@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-ec2-createtags.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowedroot@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-ec2-createdeletetags.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowedroot@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-delete-secgroup.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowedroot@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-elb-create.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowedroot@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-elb-addremovetags.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowedroot@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-elb-modifytargetgroup.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowedroot@rok-tools:~/ops/deployments# aws iam simulate-principal-policy \ > --cli-input-yaml "$(j2 rok/eks/can-i/eks-alb-iam-alb-elb-registertargets.yaml.j2)" | \ > jq -r '.EvaluationResults[].EvalDecision' | \ > sort -u allowed
What’s Next¶
The next step is to deploy AWS Load Balancer Controller.