Clean Up

To tear down an existing Rok installation on Kubernetes and reach a clean state make sure you follow the sections below. If you encounter any abnormal behavior, please take a look at our Troubleshooting FAQ which may contain useful information on how to proceed.

Uninstall Rok

This section contains needed steps to tear down an existing Rok deployment on Kubernetes and reach a clean state. If you have integrated your cluster with Kubeflow, make sure you purge it first.

Delete PVCs

Before purging Rok make sure that no PVCs of StorageClass rok are left in the cluster.

To see all PVC and whether they are in use run the following script:

$ kubectl describe pvc -A | \
>   grep -e ^Name: -e ^StorageClass: -e ^Mounted.By: -e Namespace: | \
>     cut -d: -f2 | paste - - - - | sort -k 3 | column -t

For example, in an environment with

  • Kubeflow installed on top of Rok,
  • two notebooks of the admin, and
  • three stale PVCs from a PostgreSQL used for testing Rok

you should see:

data-rok-etcd-0                rok                         gp2  rok-etcd-0
data-rok-postgresql-0          rok                         gp2  rok-postgresql-0
redis-data-rok-redis-0         rok                         gp2  rok-redis-0
authservice-pvc                istio-system                rok  authservice-0
workspace-test-jpp3x7iet       kubeflow-admin-example-com  rok  clone-0
katib-mysql                    kubeflow                    rok  katib-mysql-57884cb488-km42k
metadata-mysql                 kubeflow                    rok  metadata-db-868bb7665b-p4lht
minio-pv-claim                 kubeflow                    rok  minio-7d49d7c549-2bvqc
mysql-pv-claim                 kubeflow                    rok  mysql-645b6d9679-qphb9
data-test-postgresql-master-0  default                     rok  <none>
data-test-postgresql-slave-0   default                     rok  <none>
data-test-postgresql-slave-1   default                     rok  <none>
workspace-test-vknwungqq       kubeflow-admin-example-com  rok  test-0

Delete all pods that use PVCs of StorageClass rok and then delete the PVCs as well.

Delete RokCluster

Caution

The command listed below wipes Rok and its external services along with their data, i.e., PVCs.

Use the following command to delete your Rok installation. This includes the Rok cluster, operator and external services (etcd, Redis, etc.):

$ rok-deploy --delete install/rok

Delete Rok Registry

Use the following command to delete your Rok Registry installation. This includes the Rok Registry cluster, operator and external services (etcd, PostgreSQL, etc.):

$ rok-deploy --delete install/registry

Delete Istio

Use the following command to delete your Istio installation:

$ rok-deploy --delete install/istio

Otherwise, you can delete Istio components and CRDs but leave the istio-system namespace behind, since it might contain resources that you want to keep, e.g., Ingress istio-ingress.

To delete Istio components run:

$ rok-deploy --delete rok/rok-external-services/istio/istio-1-9/istio-install/overlays/deploy
$ rok-deploy --delete rok/rok-external-services/istio/istio-1-9/istio-crds/overlays/deploy

Purge Buckets

After you delete a Rok installation we advise you to also purge the S3 buckets which Rok had access to.

For this, you can use the rok-s3-bucket-purge CLI utility, from inside your management environment:

$ AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
$ ROK_BUCKET_PREFIX="rok-${AWS_ACCOUNT_ID?}-${AWS_DEFAULT_REGION?}-${CLUSTERNAME?}-${ROK_CLUSTER_NAMESPACE?}-${ROK_CLUSTER_NAME?}"
$ ROK_BUCKET_PREFIX=${ROK_BUCKET_PREFIX%-rok-rok}
$ rok-s3-bucket-purge --bucket-prefix ${ROK_BUCKET_PREFIX?}

Delete Cloud Identity

If you authorized Rok to access S3 resources, using rok-deploy, s3-authorize or manually using the suggested YAML template, you probably want to delete the created resources.

In order to delete all the resources created via CloudFormation, run the following command:

$ aws cloudformation delete-stack --stack-name ${STACK_NAME?}

Delete Monitoring Stack

If you deployed Rok’s monitoring stack as described in the Monitor Rok guide you will probably want to tear it down and delete its data.

To completely stop Rok’s monitoring services running on Kubernetes and purge their data you need to:

  1. Go inside the local clone of Arrikto’s deployments git repository:

    $ cd ~/ops/deployments
    
  2. Delete the core components of the monitoring stack:

    $ kubectl delete -k rok/monitoring/install/overlays/deploy
    
  3. Delete the Prometheus Operator, the monitoring namespace as well as any related CustomResourceDefintions:

    Caution

    The following command will permanently delete Prometheus’s time series database and any custom Grafana configuration.

    $ kubectl delete -k rok/monitoring/setup
    

Reconfigure Default Storage Class

During deployment we make rok the default storage class. Revert this to gp2:

$ kubectl annotate storageclass gp2 \
>    storageclass.kubernetes.io/is-default-class=true \
>    --overwrite

Delete Cloud Identity Roles

During the installation we created the following IAM roles for service accounts:

To clean up the corresponding IAM resources:

  1. Specify which IAM role to operate on:

    $ export IAM_ROLE_NAME=<role>
    
  2. Detach the attached policies:

    $ aws iam list-attached-role-policies \
    >     --role-name ${IAM_ROLE_NAME?} | \
    >         jq -r '.AttachedPolicies[].PolicyArn' | \
    >              xargs -r -n1 -I{} \
    >     aws iam detach-role-policy \
    >         --role-name ${IAM_ROLE_NAME?} \
    >         --policy-arn {}
    
  3. Delete the role:

    $ aws iam delete-role --role-name ${IAM_ROLE_NAME?}
    

Delete Node Groups

Note

This section is only for Managed node groups. For deleting Self-managed node groups use the Console.

Warning

Deleting the nodegroup means that all live data on NVMe devices will be lost. Future versions of Rok will auto-snapshot any Rok PVC so that you can recover them once a new nodegroup is added.

Warning

Deleting the nodegroup most probably means that any newer nodegroups will use a newer AMI release which may have an incompatible Kernel. Future versions of Rok will be able to auto-generate necessary modules on the fly.

Obtain the existing nodegroups of your cluster and delete them one-by-one:

$ aws eks list-nodegroups --cluster-name ${CLUSTERNAME?} | \
>   jq -r '.nodegroups[]' | \
>     xargs -r -n1 aws eks delete-nodegroup --cluster-name ${CLUSTERNAME?} --nod

Delete Kubernetes Cluster

To delete the EKS cluster, make sure you Delete Node Groups first. Then run:

$ aws eks delete-cluster --name ${CLUSTERNAME?}