Authorize Access to Object Storage on Azure

This guide will walk you through configuring Rok to have access to object storage resources on Azure.

What You'll Need

Procedure

  1. Go inside your clone of the GitOps repo:

    root@rok-tools:~# cd ~/ops/deployments
    
  2. Retrieve the Resource ID of the managed identity:

    root@rok-tools:~/ops/deployments# export IDENTITY_RESOURCE_ID="$(az identity show \
    > -g ${AZ_RESOURCE_GROUP?} -n ${AZ_MANAGED_IDENTITY?} --query id -otsv)"
    
  3. Create the namespaces Rok will use:

    root@rok-tools:~/ops/deployments# rok-deploy --apply rok/rok-namespaces/overlays/deploy
    
  4. Select the name of the Pod identity Rok will use to access Azure Blob Storage:

    root@rok-tools:~/ops/deployments# export POD_IDENTITY_NAME=rok-${ROK_CLUSTER_NAME?}
    
  5. Remove the Rok cluster name from the Pod identity name if it is equal to rok:

    root@rok-tools:~/ops/deployments# export POD_IDENTITY_NAME=${POD_IDENTITY_NAME%-rok}
    
  6. Create a Pod identity for S3Proxy in the rok namespace:

    root@rok-tools:~/ops/deployments# az aks pod-identity add \
    >  --resource-group ${AZ_RESOURCE_GROUP?} \
    >  --cluster-name ${AKS_CLUSTER?} \
    >  --namespace ${ROK_CLUSTER_NAMESPACE?} \
    >  --name ${POD_IDENTITY_NAME?} \
    >  --identity-resource-id ${IDENTITY_RESOURCE_ID?}
    

    Troubleshooting

    The command failed with a 'Bad Request' error

    Azure may take some time to enable Pod identities in a cluster. When trying to create a Pod identity before the changes are fully propagated, the following error may occur:

    Operation failed with status: 'Bad Request'. Details: Cluster identity has no assignment permission over identity 'resourceId: /subscriptions/f7a20dff-0a55-42bd-bec6-a18c6c370d0e/resourcegroups/arr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/rok - clientId: 6c5f59e0-e4f3-4ac8-939c-66cf87d8056d - objectId: 95520a25-3c7d-4b94-b516-cdc4cb2d881a'. Please grant at least 'Managed Identity Operator' permission before assigning pod identity
    

    If you see this error, wait for a few minutes for Azure to propagate permissions, and try creating the Pod identity again.

  7. Configure S3Proxy to access the Azure Storage Account:

    root@rok-tools:~/ops/deployments# j2 rok/rok-external-services/s3proxy/overlays/deploy/config.env.j2 -o rok/rok-external-services/s3proxy/overlays/deploy/config.env
    
  8. Obtain your Pod identity name and copy it to your clipboard as you are going to use this value in the following step:

    root@rok-tools:~/ops/deployments# echo ${POD_IDENTITY_NAME?}
    rok
    
  9. Edit rok/rok-external-services/s3proxy/overlays/deploy/patches/daemonset.yaml to set the aadpodidbinding label to the name of the Pod identity you created, so S3Proxy can use it to access Azure Blob Storage.

    spec:
      template:
        metadata:
          labels:
            aadpodidbinding: rok  # <-- Update this line with your Pod identity name
    
  10. Generate random credentials for Rok to access S3Proxy:

    root@rok-tools:~/ops/deployments# export S3PROXY_IDENTITY="$(openssl rand -hex 16)"
    root@rok-tools:~/ops/deployments# export S3PROXY_CREDENTIAL="$(openssl rand -hex 32)"
    
  11. Provide the generated credentials to S3Proxy:

    root@rok-tools:~/ops/deployments# j2 rok/rok-external-services/s3proxy/overlays/deploy/secrets/credentials.env.j2 -o rok/rok-external-services/s3proxy/overlays/deploy/secrets/credentials.env
    
  12. Edit rok/rok-cluster/overlays/deploy/kustomization.yaml to set the parent of the deploy kustomization overlay to aks:

    bases:
    - ../aks
    
  13. Select the bucket prefix Rok will use to store its snapshots in Azure Blob Storage:

    root@rok-tools:~/ops/deployments# S3_BUCKET_PREFIX="rok-${ROK_CLUSTER_NAMESPACE?}-${ROK_CLUSTER_NAME?}"
    
  14. Remove the Rok cluster namespace and name if they are both equal to rok:

    root@rok-tools:~/ops/deployments# export S3_BUCKET_PREFIX=${S3_BUCKET_PREFIX%-rok-rok}
    
  15. Obtain your bucket prefix and copy it to your clipboard, as you are going to use this value in the next step:

    root@rok-tools:~/ops/deployments# echo ${S3_BUCKET_PREFIX?}
    rok
    
  16. Edit rok/rok-cluster/overlays/deploy/patches/configvars.yaml to set the daemons.s3d.aws.access_key_id and daemons.s3d.aws.secret_access_key Rok Cluster configuration variables to the credentials you generated above.

    spec:
      configVars:
        daemons.s3d.bucket_prefix: "<S3_BUCKET_PREFIX>"  # <-- Update this line with your bucket prefix
        daemons.s3d.aws.access_key_id: "<S3PROXY_IDENTITY>"  # <-- Update this line with your S3Proxy Access Key ID
        daemons.s3d.aws.secret_access_key: "<S3PROXY_CREDENTIAL>"  # <-- Update this line with your S3Proxy Secret Access Key
    
  17. Track all changes in the git repository:

    root@rok-tools:~/ops/deployments# git add rok/rok-cluster rok/rok-external-services
    
  18. Commit the changes:

    root@rok-tools:~/ops/deployments# git commit -m "Configure Azure Blob Storage access for Rok"
    

Verify

  1. Verify that the Pod identity exists and that it is associated with the Rok managed identity:

    1. Specify a filter to select Rok's Pod identity:

      root@rok-tools:~# export FILTER="name=='${POD_IDENTITY_NAME?}' && \
      >    namespace=='${ROK_CLUSTER_NAMESPACE?}' && \
      >    identity.resourceId=='${IDENTITY_RESOURCE_ID?}'"
      
    2. Retrieve the Pod identity and ensure it exists:

      root@rok-tools:~# az aks pod-identity list --cluster-name ${AKS_CLUSTER?} \
      >    --resource-group ${AZ_RESOURCE_GROUP?} \
      >    --query "podIdentityProfile.userAssignedIdentities[?${FILTER?}]"
      [
        {
          "bindingSelector": null,
          "identity": {
            "clientId": "09a3e33e-4bfb-4247-afa2-2b7c453d860d",
            "objectId": "3b183c53-db99-4ea6-914d-c743b56c3d4f",
            "resourceId": "/subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourcegroups/arrikto/providers/Microsoft.ManagedIdentity/userAssignedIdentities/rok-arrikto-cluster"
          },
          "name": "rok",
          "namespace": "rok",
          "provisioningInfo": null,
          "provisioningState": "Assigned"
        }
      ]
      
    3. Verify that Azure has assigned the Pod identity to the AKS cluster, by ensuring that the value of the provisioningState field in the output of the previous step is Assigned.

  2. Verify that there are no Pod identity exceptions in your AKS cluster by ensuring that the following command produces no output:

    root@rok-tools:~# az aks pod-identity list --cluster-name ${AKS_CLUSTER?} \
    >    --resource-group ${AZ_RESOURCE_GROUP?} \
    >    --query "podIdentityProfile.userAssignedIdentityExceptions"
    
  3. Validate the AzureIdentity Kubernetes object:

    1. Ensure that an AzureIdentity object associated with the Pod identity exists in the Rok cluster namespace:

      root@rok-tools:~# kubectl get azureidentity ${POD_IDENTITY_NAME?} \
      >    -n ${ROK_CLUSTER_NAMESPACE?}
      NAME   AGE
      rok    10m
      
    2. Retrieve the clientID field of the AzureIdentity object:

      root@rok-tools:~# kubectl get azureidentity ${POD_IDENTITY_NAME?} \
      >    -n ${ROK_CLUSTER_NAMESPACE} \
      >    -o json \
      >    | jq -r .spec.clientID
      09a3e33e-4bfb-4247-afa2-2b7c453d860d
      
    3. Retrieve the clientId field of the Rok managed identity:

      root@rok-tools:~# az identity show \
      >    --resource-group ${AZ_RESOURCE_GROUP?} \
      >    --name ${AZ_MANAGED_IDENTITY?} \
      >    --query clientId \
      >    --output tsv
      09a3e33e-4bfb-4247-afa2-2b7c453d860d
      
    4. Verify that the clientID field of the AzureIdentity object matches the clientId field of the managed identity.

    5. Retrieve the resourceID field of the AzureIdentity object:

      root@rok-tools:~# kubectl get azureidentity ${POD_IDENTITY_NAME?} \
      >    -n ${ROK_CLUSTER_NAMESPACE} \
      >    -o json \
      >    | jq -r .spec.resourceID
      /subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourcegroups/arrikto/providers/Microsoft.ManagedIdentity/userAssignedIdentities/rok-arrikto-cluster
      
    6. Retrieve the id field of the Rok managed identity:

      root@rok-tools:~# az identity show \
      >    --resource-group ${AZ_RESOURCE_GROUP?} \
      >    --name ${AZ_MANAGED_IDENTITY?} \
      >    --query id \
      >    --output tsv
      /subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourcegroups/arrikto/providers/Microsoft.ManagedIdentity/userAssignedIdentities/rok-arrikto-cluster
      
    7. Verify that the resourceID field of the AzureIdentity object matches the id field of the managed identity.

  4. Validate the AzureIdentityBinding Kubernetes object:

    1. Ensure that an AzureIdentityBinding object associated with the Pod identity exists in the Rok cluster namespace:

      root@rok-tools:~# kubectl get azureidentitybinding ${POD_IDENTITY_NAME?}-binding \
      >    -n ${ROK_CLUSTER_NAMESPACE?}
      NAME           AGE
      rok-binding    10m
      
    2. Retrieve the azureIdentity field of the AzureIdentityBinding object:

      root@rok-tools:~# kubectl get azureidentitybinding ${POD_IDENTITY_NAME?}-binding \
      >    -n ${ROK_CLUSTER_NAMESPACE} \
      >    -o json \
      >    | jq -r .spec.azureIdentity
      rok
      
    3. Retrieve the selector field of the AzureIdentityBinding object:

      root@rok-tools:~# kubectl get azureidentitybinding ${POD_IDENTITY_NAME?}-binding \
      >    -n ${ROK_CLUSTER_NAMESPACE} \
      >    -o json \
      >    | jq -r .spec.selector
      rok
      
    4. Find the name of the AzureIdentity object:

      root@rok-tools:~# echo ${POD_IDENTITY_NAME?}
      rok
      
    5. Verify that the azureIdentity and selector fields of the AzureIdentityBinding object match the name of the AzureIdentity object.

  5. Verify that you can authorize a request to the Rok storage account using this Pod identity:

    1. Go inside your clone of the GitOps repo:

      root@rok-tools:~# cd ~/ops/deployments
      
    2. Render the rok-verify-storage-access-aks manifest template:

      root@rok-tools:~/ops/deployments# j2 rok/rok-test/rok-verify-storage-access-aks.yaml.j2 -o rok/rok-test/rok-verify-storage-access-aks.yaml
      

      Alternatively, save the rok-verify-storage-access-aks manifest template provided below or download rok-verify-storage-access-aks.yaml.j2 and use it locally.

    3. Track the rendered rok-verify-storage-access-aks manifest:

      root@rok-tools:~/ops/deployments# git add rok/rok-test/rok-verify-storage-access-aks.yaml
      
    4. Commit the changes:

      root@rok-tools:~/ops/deployments# git commit -m "Configure Pod to verify storage access"
      
    5. Deploy the rendered rok-verify-storage-access-aks manifest to your AKS cluster:

      root@rok-tools:~/ops/deployments# kubectl apply -f rok/rok-test/rok-verify-storage-access-aks.yaml
      pod/rok-verify-storage-access created
      
    6. Wait for the status of the rok-verify-storage-access Pod to become Running:

      root@rok-tools:~/ops/deployments# kubectl get pods rok-verify-storage-access -n ${ROK_CLUSTER_NAMESPACE?}
      NAME                        READY   STATUS    RESTARTS   AGE
      rok-verify-storage-access   1/1     Running   0          20s
      
    7. Verify that the Rok managed identity has been assigned to the Virtual Machine Scale Set (VMSS) of the node where this Pod is running:

      1. Find the node resource group of the user node pool:

        root@rok-tools:~# export AZ_NODE_RESOURCE_GROUP=$(az aks show -o tsv \
        >     --resource-group ${AZ_RESOURCE_GROUP?} \
        >     --name ${AKS_CLUSTER?} \
        >     --query nodeResourceGroup)
        
      2. Find the VMSS in the node resource group that corresponds to the user node pool:

        root@rok-tools:~# export VMSS_NAME=$(az resource list -o tsv \
        >     --resource-type Microsoft.Compute/virtualMachineScaleSets \
        >     --resource-group ${AZ_NODE_RESOURCE_GROUP?} \
        >     --query "[?tags.poolName=='workers'].name")
        
      3. Find the identities assigned to the VMSS:

        root@rok-tools:~# az vmss show \
        >    --name ${VMSS_NAME?} \
        >    --resource-group ${AZ_NODE_RESOURCE_GROUP?} \
        >    --query "keys(identity.userAssignedIdentities)" \
        >    --output tsv
        /subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourceGroups/MC_arrikto_arrikto-cluster_eastus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/arrikto-cluster-agentpool
        /subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourcegroups/MC_arrikto_arrikto-cluster_eastus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-arrikto-cluster
        /subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourcegroups/arrikto/providers/microsoft.managedidentity/userassignedidentities/rok-arrikto-cluster
        
      4. Retrieve the Rok managed identity:

        root@rok-verify-storage-access:~# echo ${IDENTITY_RESOURCE_ID?}
        /subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourcegroups/arrikto/providers/Microsoft.ManagedIdentity/userAssignedIdentities/rok-arrikto-cluster
        
      5. Verify that the Rok managed identity is in the above list.

    8. Enter the Pod:

      root@rok-tools:~/ops/deployments# kubectl exec -ti rok-verify-storage-access -n ${ROK_CLUSTER_NAMESPACE?} -- /bin/bash
      
    9. Log in with the service principal of the Pod identity:

      root@rok-verify-storage-access:~# az login --identity
      [
        {
          "environmentName": "AzureCloud",
          "homeTenantId": "2bf793e0-71b3-4a6c-ba8a-96a6755de088",
          "id": "3b63afce-113a-4798-a303-f37dada04319",
          "isDefault": true,
          "managedByTenants": [],
          "name": "arrikto-dev",
          "state": "Enabled",
          "tenantId": "2bf793e0-71b3-4a6c-ba8a-96a6755de088",
          "user": {
            "assignedIdentityInfo": "MSI",
            "name": "systemAssignedIdentity",
            "type": "servicePrincipal"
          }
        }
      ]
      

      Troubleshooting

      The command failed with 'MSI endpoint is not responding'

      If the above command failed with the following error:

      MSI endpoint is not responding. Please make sure MSI is configured correctly.
      Error detail: MSI: Failed to acquire tokens after 12 times
      

      then do the following:

      1. Try to obtain an OAuth access token from the Azure instance metadata service:

        root@rok-verify-storage-access:~# curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com/' -H "Metadata: true" -i
        HTTP/1.1 404 Not Found
        Content-Type: text/plain; charset=utf-8
        X-Content-Type-Options: nosniff
        Date: Fri, 06 Aug 2021 19:53:22 GMT
        Content-Length: 46
        
        no azure identity found for request clientID
        
      2. Inspect the output of the above command. If it failed with the no azure identity found for request clientID message, then either the aadpodidbinding label is wrong (i.e., it does not refer to Rok's Pod identity), or it does not exist at all. In this case, go back to the Procedure section and repeat all steps.

    10. Verify that the returned OAuth token belongs to the Rok managed identity:

      1. Get an OAuth token from the Azure instance metadata service, and parse the managed identity that it belongs to:

        root@rok-verify-storage-access:~# curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com/' -H "Metadata: true" -s | cut -d'.' -f2 | base64 -d | jq -r .xms_mirid
        /subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourcegroups/arrikto/providers/Microsoft.ManagedIdentity/userAssignedIdentities/rok-arrikto-cluster
        
      2. Retrieve the Rok managed identity:

        root@rok-verify-storage-access:~# echo ${IDENTITY_RESOURCE_ID?}
        /subscriptions/3b63afce-113a-4798-a303-f37dada04319/resourcegroups/arrikto/providers/Microsoft.ManagedIdentity/userAssignedIdentities/rok-arrikto-cluster
        
      3. Ensure that these two identities match.

    11. List the blobs in a non-existent container in the storage account:

      root@rok-verify-storage-access:~# az storage blob list \
      >    --account-name ${STORAGE_ACCOUNT_NAME?} \
      >    --container-name ${BUCKET_PREFIX?}-chocks-ca \
      >    --auth-mode login
      The specified container does not exist.
      RequestId:8ec2d64b-e01e-002d-6aa1-8af874000000
      Time:2021-08-06T09:02:35.4218807Z
      ErrorCode:ContainerNotFound
      Error:None
      
    12. Ensure that the above command failed with a ContainerNotFound error. This means that you have successfully authorized a request using the Pod identity.

      Troubleshooting

      The command failed due to insufficient permissions

      If the above command failed with the following error message:

      You do not have the required permissions needed to perform this operation.
      Depending on your operation, you may need to be assigned one of the following roles:
          "Storage Blob Data Contributor"
          "Storage Blob Data Reader"
          "Storage Queue Data Contributor"
          "Storage Queue Data Reader"
      
      If you want to use the old authentication method and allow querying for the right account key, please use the "--auth-mode" parameter and "key" value.
      

      then the Rok managed identity does not have the necessary permissions to perform this API call.

      Make sure you have followed the instructions in the Create Cloud Identity section.

    13. Return to your management environment:

      root@rok-verify-storage-access:~# exit
      
    14. Delete the rok-verify-storage-access Pod:

      root@rok-tools:~/ops/deployments# kubectl delete -f rok/rok-test/rok-verify-storage-access-aks.yaml
      pod "rok-verify-storage-access" deleted
      

Summary

You have successfully provided Rok with access to the object storage service of your cloud provider.

What's Next

The next step is to grant Rok access to Arrikto's private container registry, so that it can pull images from it.