Authorize Access to Object Storage on AWS

This guide will walk you through configuring Rok to have access to object storage resources on AWS.

Fast Forward

If you have already configured Rok to have access to object storage, expand this box to fast-forward.

  1. Proceed to the Verify section.

Choose one of the following options to give Rok access to the object storage service:

Air Gapped

Follow Option 2 and proceed with the manual installation.

What You’ll Need

Option 1: Authorize Access to Object Storage Automatically (preferred)

Authorize Rok to access the object storage service by following the on-screen instructions on the rok-deploy user interface.

If rok-deploy is not already running, start it with:

root@rok-tools:~# rok-deploy --run-from objectstorage
../../../_images/objectstorage.png

Proceed to the Summary section.

Option 2: Authorize Access to Object Storage Manually

If you want to authorize Rok to access the object storage service manually, follow the instructions below.

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments

  2. Restore the required context from previous sections:

    root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,cloudidentity})
    root@rok-tools:~/ops/deployments# export AWS_ACCOUNT_ID AWS_DEFAULT_REGION \ > AWS_S3_BUCKET_PREFIX AWS_S3_ROLE

  3. Configure Rok to use the IAM role in order to access S3 storage:

    root@rok-tools:~/ops/deployments# j2 \ > rok/rok-cluster/overlays/deploy/patches/storage.yaml.j2 \ > -o rok/rok-cluster/overlays/deploy/patches/storage.yaml

  4. Configure Rok to use the S3 bucket prefix you specified:

    root@rok-tools:~/ops/deployments# j2 \ > rok/rok-cluster/overlays/deploy/patches/configvars.yaml.j2 \ > -o rok/rok-cluster/overlays/deploy/patches/configvars.yaml

    Important

    Rok will create a number of buckets with this specific prefix. Please note that Rok assumes it owns all buckets with names starting with this prefix, for example, for garbage collection purposes. You must not share this prefix with any other application.

  5. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit -am "Authorize Access to Object Storage on AWS"

  6. Mark your progress:

    root@rok-tools:~/ops/deployments# export DATE=$(date -u "+%Y-%m-%dT%H.%M.%SZ")
    root@rok-tools:~/ops/deployments# git tag -a deploy/${DATE?}/release-1.5/objectstorage \ > -m "Authorize Access to Object Storage on AWS"

Verify

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments

  2. Restore the required context from previous sections:

    root@rok-tools:~/ops/deployments# source <(cat deploy/env.{envvars-aws,\ > cloudidentity,eks-identity})
    root@rok-tools:~/ops/deployments# export AWS_ACCOUNT_ID AWS_DEFAULT_REGION \ > ROK_CLUSTER_NAMESPACE ROK_CLUSTER_NAME \ > AWS_S3_BUCKET_PREFIX AWS_S3_ROLE EKS_CLUSTER_OIDC

  3. Build the manifests and make sure the output contains four lines:

    root@rok-tools:~/ops/deployments# kustomize build rok/rok-cluster/overlays/deploy | \ > grep -o -e "daemons.s3d.bucket_prefix: ${AWS_S3_BUCKET_PREFIX?}" \ > -e "AWSRoleARN: arn:aws:iam::${AWS_ACCOUNT_ID?}:role/${AWS_S3_ROLE?}" \ > -e "endpoint: https://s3.${AWS_DEFAULT_REGION?}.amazonaws.com" \ > -e "region: ${AWS_DEFAULT_REGION?}" daemons.s3d.bucket_prefix: rok-123451234512-us-west-2-arrikto-cluster AWSRoleARN: arn:aws:iam::123451234512:role/rok-us-west-2-arrikto-cluster endpoint: https://s3.us-west-2.amazonaws.com region: us-west-2

  4. Deploy a Pod using the service account of Rok:

    1. Create the namespaces Rok will use:

      root@rok-tools:~/ops/deployments# rok-deploy --apply rok/rok-namespaces/overlays/deploy

    2. Render the rok-verify-service-account-eks manifest template:

      root@rok-tools:~/ops/deployments# j2 rok/rok-test/rok-verify-service-account-eks.yaml.j2 \ > -o rok/rok-test/rok-verify-service-account-eks.yaml

      Alternatively, download the rok-verify-service-account-eks manifest template provided below and use it locally.

      rok-verify-service-account-eks.yaml.j2
      Download file
      1# This file is part of Rok.
      2#
      3# Copyright © 2021 Arrikto Inc. All Rights Reserved.
      4-10
      4
      5apiVersion: v1
      6kind: ServiceAccount
      7metadata:
      8 labels:
      9 app.kubernetes.io/part-of: rok-verify
      10 annotations:
      11 eks.amazonaws.com/role-arn: arn:aws:iam::{{AWS_ACCOUNT_ID}}:role/{{AWS_S3_ROLE}}
      12 name: {{ROK_CLUSTER_NAME}}
      13 namespace: {{ROK_CLUSTER_NAMESPACE}}

    3. Render the rok-verify-storage-access-eks manifest template:

      root@rok-tools:~/ops/deployments# j2 rok/rok-test/rok-verify-storage-access-eks.yaml.j2 \ > -o rok/rok-test/rok-verify-storage-access-eks.yaml

      Alternatively, download the rok-verify-storage-access-eks manifest template provided below and use it locally.

      rok-verify-storage-access-eks.yaml.j2
      Download file
      1# This file is part of Rok.
      2#
      3# Copyright © 2021 Arrikto Inc. All Rights Reserved.
      4-26
      4
      5apiVersion: v1
      6kind: Pod
      7metadata:
      8 name: rok-verify-storage-access
      9 namespace: {{ROK_CLUSTER_NAMESPACE}}
      10 labels:
      11 app: rok-verify-storage-access
      12spec:
      13 serviceAccountName: {{ROK_CLUSTER_NAME}}
      14 containers:
      15 - name: rok-verify-storage-access
      16 image: gcr.io/arrikto/rok-tools:release-1.5-l0-release-1.5.3
      17 imagePullPolicy: IfNotPresent
      18 workingDir: /root
      19 env:
      20 - name: AWS_S3_BUCKET_PREFIX
      21 value: "{{AWS_S3_BUCKET_PREFIX}}"
      22 - name: EKS_CLUSTER_OIDC
      23 value: "{{EKS_CLUSTER_OIDC}}"
      24 - name: ROK_CLUSTER_NAME
      25 value: "{{ROK_CLUSTER_NAME}}"
      26 - name: ROK_CLUSTER_NAMESPACE
      27 value: "{{ROK_CLUSTER_NAMESPACE}}"
      28 - name: AWS_DEFAULT_REGION
      29 value: "{{AWS_DEFAULT_REGION}}"

      Air Gapped

      Edit the rendered manifest rok/rok-test/rok-verify-storage-access-eks.yaml to use the mirrored rok-tools image from your internal registry:

      containers: - name: rok-verify-storage-access image: gcr.io/arrikto/rok-tools:release-1.5-l0-release-1.5.3 # <-- Edit this line to point to your mirrored rok-tools image

    4. Commit the changes:

      root@rok-tools:~/ops/deployments# git commit -am "Configure Pod and SA to verify storage access"

    5. Create the rok service account:

      root@rok-tools:~/ops/deployments# kubectl create -f rok/rok-test/rok-verify-service-account-eks.yaml serviceaccount/rok created

      Troubleshooting

      The service account already exists

      If you have already deployed Rok and created its service account, then the above command will fail with the following error:

      Error from server (AlreadyExists): error when creating "rok/rok-test/rok-verify-service-account-eks.yaml": serviceaccounts "rok" already exists

      In this case, ignore this error and proceed with the verification steps below.

    6. Deploy the rendered rok-verify-storage-access-eks manifest to your EKS cluster:

      root@rok-tools:~/ops/deployments# kubectl apply -f rok/rok-test/rok-verify-storage-access-eks.yaml pod/rok-verify-storage-access created

    7. Wait for the rok-verify-storage-access Pod to come up:

      root@rok-tools:~/ops/deployments# kubectl get pods rok-verify-storage-access -n ${ROK_CLUSTER_NAMESPACE?} NAME READY STATUS RESTARTS AGE rok-verify-storage-access 1/1 Running 0 20s

  5. Verify the spec of the Pod:

    1. Inspect the Pod spec and verify that the EKS Pod identity admission controller has extended it with the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables, as well as an aws-iam-token projected service account token:

      root@rok-tools:~/ops/deployments# kubectl get pod rok-verify-storage-access -n ${ROK_CLUSTER_NAMESPACE?} -o yaml apiVersion: v1 kind: Pod [...] spec: containers: - env: [...] - name: AWS_ROLE_ARN value: arn:aws:iam::123456789012:role/rok-us-west-2-arrikto-cluster - name: AWS_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token [...] volumeMounts: [...] - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount name: aws-iam-token readOnly: true [...] volumes: - name: aws-iam-token projected: defaultMode: 420 sources: - serviceAccountToken: audience: sts.amazonaws.com expirationSeconds: 86400 path: token [...]

    2. Compute the ARN of the IAM role you created for Rok:

      root@rok-tools:~/ops/deployments# echo "arn:aws:iam::${AWS_ACCOUNT_ID?}:role/${AWS_S3_ROLE?}" arn:aws:iam::123456789012:role/rok-us-west-2-arrikto-cluster

    3. Verify that the ARN of the IAM role you created for Rok matches the value of the AWS_ROLE_ARN environment variable in the Pod spec.

    4. Verify that the audience field of the projected service account token of the Pod spec is sts.amazonaws.com.

  6. Verify the Pod credentials:

    1. Enter the Pod:

      root@rok-tools:~/ops/deployments# kubectl exec -ti rok-verify-storage-access \ > -n ${ROK_CLUSTER_NAMESPACE?} -- /bin/bash

    2. Decode and inspect the web identity token projected into the Pod:

      root@rok-verify-storage-access:~# cat ${AWS_WEB_IDENTITY_TOKEN_FILE?} | cut -d'.' -f2 | base64 -d | jq { "aud": [ "sts.amazonaws.com" ], "exp": 1630400784, "iat": 1630314384, "iss": "https://oidc.eks.us-west-2.amazonaws.com/id/C4384257372B0F97FEC2C5A47EF6EDAB", "kubernetes.io": { "namespace": "rok", "pod": { "name": "rok-verify-storage-access", "uid": "d835a896-54be-489e-ba6c-99d90c32ebbe" }, "serviceaccount": { "name": "rok", "uid": "a0068c4b-05a1-42a8-9a0e-03235280e7f6" } }, "nbf": 1630314384, "sub": "system:serviceaccount:rok:rok" }

    3. Verify that the aud field of the web identity token is sts.amazonaws.com.

    4. Find the URL of the OIDC provider of the EKS cluster:

      root@rok-verify-storage-access:~# echo "https://${EKS_CLUSTER_OIDC?}" https://oidc.eks.us-west-2.amazonaws.com/id/C4384257372B0F97FEC2C5A47EF6EDAB

    5. Verify that the iss field of the web identity token matches the URL of the OIDC provider of the EKS cluster.

    6. Find the Rok cluster namespace and name:

      root@rok-verify-storage-access:~# echo ${ROK_CLUSTER_NAMESPACE?} rok root@rok-verify-storage-access:~# echo ${ROK_CLUSTER_NAME?} rok

    7. Verify that the kubernetes.io.namespace field of the web identity token matches the Rok cluster namespace.

    8. Verify that the kubernetes.io.pod.name field of the web identity token is rok-verify-storage-access.

    9. Verify that the kubernetes.io.serviceaccount.name field of the web identity token matches the Rok cluster name.

    10. Compute the Kubernetes username for the Rok service account:

      root@rok-verify-storage-access:~# echo system:serviceaccount:${ROK_CLUSTER_NAMESPACE?}:${ROK_CLUSTER_NAME?} system:serviceaccount:rok:rok

    11. Verify that the sub field of the web identity token matches the username of the Rok service account.

  7. Perform a request towards S3:

    1. List versions in a Rok S3 bucket:

      root@rok-verify-storage-access:~# aws s3api list-object-versions \ > --bucket ${AWS_S3_BUCKET_PREFIX?}-chocks-ca An error occurred (NoSuchBucket) when calling the ListObjectVersions operation: The specified bucket does not exist

      Troubleshooting

      The AssumeRoleWithWebIdentity operation failed with AccessDenied

      If the above command failed with the following error:

      An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

      then this means that you have not authorized the Pod’s web identity to assume the IAM role for Rok. Make sure you have followed the Create Cloud Identity section to set up the policies for the IAM role you created for Rok.

      The ListObjectVersions operation failed with AccessDenied

      If the above command failed with the following error:

      An error occurred (AccessDenied) when calling the ListObjectVersions operation: Access Denied

      then this means that the Pod can assume the IAM role for Rok with its web identity, but the role does not have sufficient permissions to perform the ListObjectVersions call. In this case, do the following:

      1. Make sure you are using the bucket prefix specified in the Option 2: Authorize Access to Object Storage Manually section, and not a custom one.
      2. Make sure you have followed the Create Cloud Identity section to set up the policies for the IAM role you created for Rok.

    2. Check the output of the above command. If the output is empty, or it lists some versions, this means that you have successfully assumed the IAM role for Rok and authorized a request with it. In this case, proceed with the next step. Otherwise, if the command failed with NoSuchBucket, run the following extra steps:

      1. Create the Rok S3 bucket:

        root@rok-verify-storage-access:~# export BUCKET_CONFIGURATION="--create-bucket-configuration LocationConstraint=${AWS_DEFAULT_REGION?}"
        root@rok-verify-storage-access:~# [[ $AWS_DEFAULT_REGION == "us-east-1" ]] \ > && export BUCKET_CONFIGURATION=""
        root@rok-verify-storage-access:~# aws s3api create-bucket \ > --bucket ${AWS_S3_BUCKET_PREFIX?}-chocks-ca \ > --region ${AWS_DEFAULT_REGION?} \ > ${BUCKET_CONFIGURATION?} { "Location": "http://rok-123451234512-us-west-2-arrikto-cluster-chocks-ca.s3.amazonaws.com/" }

        Troubleshooting

        The AssumeRoleWithWebIdentity operation failed with AccessDenied

        If the above command failed with the following error:

        An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

        then this means that you have not authorized the Pod’s web identity to assume the IAM role for Rok. Make sure you have followed the Create Cloud Identity section to set up the policies for the IAM role you created for Rok.

        The CreateBucket operation failed with AccessDenied

        If the above command failed with the following error:

        An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied

        then this means that the Pod can assume the IAM role for Rok with its web identity, but the role does not have sufficient permissions to perform the CreateBucket call. In this case, do the following:

        1. Make sure you are using the bucket prefix specified in the Option 2: Authorize Access to Object Storage Manually section, and not a custom one.
        2. Make sure you have followed the Create Cloud Identity section to set up the policies for the IAM role you created for Rok.

      2. List versions in the Rok S3 bucket:

        root@rok-verify-storage-access:~# aws s3api list-object-versions \ > --bucket ${AWS_S3_BUCKET_PREFIX?}-chocks-ca

        Troubleshooting

        The AssumeRoleWithWebIdentity operation failed with AccessDenied

        If the above command failed with the following error:

        An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

        then this means that you have not authorized the Pod’s web identity to assume the IAM role for Rok. Make sure you have followed the Create Cloud Identity section to set up the policies for the IAM role you created for Rok.

        The ListObjectVersions operation failed with AccessDenied

        If the above command failed with the following error:

        An error occurred (AccessDenied) when calling the ListObjectVersions operation: Access Denied

        then this means that the Pod can assume the IAM role for Rok with its web identity, but the role does not have sufficient permissions to perform the ListObjectVersions call. In this case, do the following:

        1. Make sure you are using the bucket prefix specified in the Option 2: Authorize Access to Object Storage Manually section, and not a custom one.
        2. Make sure you have followed the Create Cloud Identity section to set up the policies for the IAM role you created for Rok.

      3. Ensure that the output of the above command is empty. This means that you have successfully assumed the IAM role for Rok and authorized a request with it.

  8. Clean up the Pod:

    1. Return to your management environment:

      root@rok-verify-storage-access:~# exit

    2. Delete the Pod:

      root@rok-tools:~/ops/deployments# kubectl delete -f rok/rok-test/rok-verify-storage-access-eks.yaml pod "rok-verify-storage-access" deleted

    3. Delete the Rok service account, if you created it in this Verify section:

      root@rok-tools:~/ops/deployments# kubectl delete sa -n ${ROK_CLUSTER_NAMESPACE} \ > -l "app.kubernetes.io/part-of=rok-verify" serviceaccount "rok" deleted

Summary

You have successfully provided Rok with access to the object storage service of your cloud provider.

What’s Next

The next step is to grant Rok access to Arrikto’s private container registry, so that it can pull images from it.

PreviousNext