Cleanup

To tear down an existing Rok installation on Kubernetes and reach a clean state make sure you follow the sections below. If you encounter any abnormal behavior, please take a look at our Troubleshooting FAQ which may contain useful information on how to proceed.

Rok

This section contains needed steps to tear down an existing Rok deployment on Kubernetes and reach a clean state. If you have integrated your cluster with Kubeflow, make sure you purge it first.

PVC

Before purging Rok make sure that no PVCs of StorageClass rok are left in the cluster.

To see all PVC and whether they are in use run the following script:

$ kubectl describe pvc -A | \
>   grep -e ^Name: -e ^StorageClass: -e ^Mounted.By: -e Namespace: | \
>     cut -d: -f2 | paste - - - - | sort -k 3 | column -t

For example, in an environment with

  • Kubeflow installed on top of Rok,
  • two notebooks of the admin, and
  • three stale PVCs from a PostgreSQL used for testing Rok

you should see:

data-rok-etcd-0                rok                         gp2  rok-etcd-0
data-rok-postgresql-0          rok                         gp2  rok-postgresql-0
redis-data-rok-redis-0         rok                         gp2  rok-redis-0
authservice-pvc                istio-system                rok  authservice-0
workspace-test-jpp3x7iet       kubeflow-admin-example-com  rok  clone-0
katib-mysql                    kubeflow                    rok  katib-mysql-57884cb488-km42k
metadata-mysql                 kubeflow                    rok  metadata-db-868bb7665b-p4lht
minio-pv-claim                 kubeflow                    rok  minio-7d49d7c549-2bvqc
mysql-pv-claim                 kubeflow                    rok  mysql-645b6d9679-qphb9
data-test-postgresql-master-0  default                     rok  <none>
data-test-postgresql-slave-0   default                     rok  <none>
data-test-postgresql-slave-1   default                     rok  <none>
workspace-test-vknwungqq       kubeflow-admin-example-com  rok  test-0

Delete all pods that use PVCs of StorageClass rok and then delete the PVCs as well.

RokCluster

Caution

The command listed below wipes Rok and its external services along with their data, i.e., PVCs.

Use the following command to delete your Rok installation. This includes the Rok cluster, operator and external services (etcd, Istio, Redis, etc.):

$ rok-deploy --delete install/rok

S3 Buckets

After you delete a Rok installation we advise you to also purge the S3 buckets which Rok had access to.

For this, you can use the rok-s3-bucket-purge CLI utility, from inside your management environment:

$ rok-s3-bucket-purge \
>     --bucket-prefix rok-${AWS_DEFAULT_REGION?}-${CLUSTERNAME?}-${ROKNAMESPACE?}-${ROKNAME?} --rok-only

IAM resources

If you authorized Rok to access S3 resources, using rok-deploy, s3-authorize or manually using the suggested YAML template, you probably want to delete the created resources.

In order to delete all the resources created via CloudFormation, run the following command:

$ aws cloudformation delete-stack --stack-name ${STACK_NAME?}

Monitoring Stack

If you deployed Rok’s monitoring stack as described in the Monitoring guide you will probably want to tear it down and delete its data.

To completely stop Rok’s monitoring services running on Kubernetes and purge their data you need to:

  1. Go inside the local clone of Arrikto’s deployments git repository:

    $ cd ~/ops/deployments
    
  2. Delete the core components of the monitoring stack:

    $ kubectl delete -k rok/monitoring/install/overlays/deploy
    
  3. Delete the Prometheus Operator, the monitoring namespace as well as any related CustomResourceDefintions:

    Caution

    The following command will permanently delete Prometheus’s time series database and any custom Grafana configuration.

    $ kubectl delete -k rok/monitoring/setup
    

Default Storage Class

During deployment we make rok the default storage class. Revert this to gp2:

$ kubectl annotate storageclass gp2 \
>    storageclass.kubernetes.io/is-default-class=true \
>    --overwrite

Istio

Here we delete Istio components and CRDs but leave istio-system namespace behind since it might contain resources that we want to keep, e.g., Ingress istio-ingress.

To delete Istio components run:

$ rok-deploy --delete rok/rok-external-services/istio/istio-1-5-7/istio-install-1-5-7/overlays/deploy
$ rok-deploy --delete rok/rok-external-services/istio/istio-1-5-7/istio-crds-1-5-7/overlays/deploy

IAM roles

During the installation we created the following IAM roles for service accounts:

To clean up the corresponding IAM resources:

  1. Specify which IAM role to operate on:

    $ export IAM_ROLE_NAME=<role>
    
  2. Detach the attached policies:

    $ aws iam list-attached-role-policies \
    >     --role-name ${IAM_ROLE_NAME?} | \
    >         jq -r '.AttachedPolicies[].PolicyArn' | \
    >              xargs -r -n1 -I{} \
    >     aws iam detach-role-policy \
    >         --role-name ${IAM_ROLE_NAME?} \
    >         --policy-arn {}
    
  3. Delete the role:

    $ aws iam delete-role --role-name ${IAM_ROLE_NAME?}
    

EKS node groups

Note

This section is only for Managed node groups. For deleting Self-managed node groups use the Console.

Warning

Deleting the nodegroup means that all live data on NVMe devices will be lost. Future versions of Rok will auto-snapshot any Rok PVC so that you can recover them once a new nodegroup is added.

Warning

Deleting the nodegroup most probably means that any newer nodegroups will use a newer AMI release which may have an incompatible Kernel. Future versions of Rok will be able to auto-generate necessary modules on the fly.

Obtain the existing nodegroups of your cluster and delete them one-by-one:

$ aws eks list-nodegroups --cluster-name ${CLUSTERNAME?} | \
>   jq -r '.nodegroups[]' | \
>     xargs -r -n1 aws eks delete-nodegroup --cluster-name ${CLUSTERNAME?} --nod

EKS cluster

To delete the EKS cluster, make sure you delete any managed EKS node groups first. Then run:

$ aws eks delete-cluster --name ${CLUSTERNAME?}