Update Retention Policy of all Buckets

This guide will walk you through updating the version retention policy of existing buckets in a Rok API installation to the default options you have configured for new buckets in guide Configure Default Retention Policy for New Buckets.

Note

This guide’s purpose is to peform a bulk update of multiple version retention policies across multiple accounts. If you wish to configure the version retention policy of a single bucket, you can also do it via the Rok UI or CLI by following the Snapshot policies for Backup guide.

What You’ll Need

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Specify the accounts in which you wish to update version retention policies:

    root@rok-tools:~# export ACCOUNTS=<ACCOUNTS>

    Replace <ACCOUNTS> with a comma-separated list of account names in which to update version retention policies. Set to ALL to apply changes to all accounts:

    root@rok-tools:~# export ACCOUNTS=ALL
  3. Specify the accounts to exclude from the operation:

    root@rok-tools:~# export EXCLUDE_ACCOUNTS=<ACCOUNTS>

    Replace <ACCOUNTS> with a comma-separated list of account names to exclude. Leave empty to apply to all accounts selected in the previous step:

    root@rok-tools:~# export EXCLUDE_ACCOUNTS=
  4. Specify the buckets of which you wish to update the version retention policy:

    root@rok-tools:~# export BUCKETS=<BUCKETS>

    Replace <BUCKETS> with a comma-separated list of bucket names to update the version retention policy of. Set to ALL to apply to all buckets:

    root@rok-tools:~# export BUCKETS=ALL
  5. Specify the buckets to exclude from the operation:

    root@rok-tools:~# export EXCLUDE_BUCKETS=<BUCKETS>

    Replace <BUCKETS> with a comma-separated list of bucket names to exclude. You must always include the notebooks and auto-backup buckets in this list, to retain the version retention policy settings that you have configured via your EKF manifests:

    root@rok-tools:~# export EXCLUDE_BUCKETS=notebooks,auto-backup

    Important

    EKF manifests already contain RokBucketConfiguration custom resources for the version retention policies of the notebooks and auto-backup buckets. This means that even if you modify these policies as part of this guide, the operator will automatically revert the change.

    However, it is still possible that a new task will be created before the Rok operator has time to revert the change. This can lead to versions being deleted in these buckets, which especially in the case of the notebooks bucket may lead to malfunctioning Kubeflow Pipelines. Therefore, we strongly recommend to always exclude the notebooks and auto-backup buckets when running this management command.

  6. Optional

    By default the command will only affect version retention policies with out-of-the-box settings, i.e., no schedule and a single retention rule to retain all versions forever. If you wish to instead affect all policies, even if they have a custom configuration, specify the following setting:

    root@rok-tools:~# export EXTRA_OPTIONS=--overwrite
  7. Run the command to update the version retention policies. Inspect the buckets that will be affected and rules that will be applied, and type y, <enter> to proceed with the update:

    root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok-gw-manage retention-policy-update \ > --accounts "${ACCOUNTS?}" --exclude-accounts "${EXCLUDE_ACCOUNTS?}" \ > --buckets "${BUCKETS?}" --exclude-buckets "${EXCLUDE_BUCKETS?}" \ > ${EXTRA_OPTIONS} 2022-09-14T17:00:09.002161+0000 api pid=3032/tid=3032/pytid=140096893715904 cache:178 [INFO] Initialized authentication backends: AuthService, KubernetesToken ------------------------------------------------------------------------- Policy ID Account Bucket ------------------------------------------------------------------------- 2e1733e8-9543-4866-aa0d-29cef79ac3c1 kubeflow-user open-vaccine 6a9ac4fe-c649-4802-b15b-7838f3b690f0 kubeflow-user titanic ------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------- Default Version Retention Rules -------------------------------------------------------------------------------------------------------- Retain the current version of each object forever Retain all versions of each object for 1 week Retain one version of each object every 1 week (starting at 2022-09-12T00:00:00+00:00) for 6 months Retain one version of each object every 3 months (starting at 2022-09-01T00:00:00+00:00) for 5 years -------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- Default Version Retention Schedule ---------------------------------------------------------- Run every 1 day, starting at 2022-09-14T00:00:00+00:00 ---------------------------------------------------------- This command will update the retention policies displayed above to use the default retention rules and schedule of the Rok Gateway. You can modify the Gateway's default rules and schedule by updating the VERSION_RETENTION_DEFAULT_RULES andVERSION_RETENTION_DEFAULT_SCHEDULE settings in '/etc/rok/gw/backend.conf.py'. Apply these changes? [y/N]: y 2022-09-14T17:05:20.991803+0000 api pid=3667/tid=3667/pytid=140560480060864 retention-policy-update:135 [INFO] Successfully updated 2 policies

Verify

  1. Specify the namespace in which to verify the version retention policy of a bucket:

    root@rok-tools:~# export NAMESPACE=<NAMESPACE>

    Replace NAMESPACE with the name of the namespace that you wish to verify:

    root@rok-tools:~# export NAMESPACE=kubeflow-user
  2. Specify the name of the bucket to verify:

    root@rok-tools:~# export BUCKET_NAME=<BUCKET_NAME>

    Replace BUCKET_NAME with the name of the bucket you wish to verify. Pick an existing bucket outside of the list of buckets you excluded above, for example:

    root@rok-tools:~# export BUCKET_NAME=verify-retention
  3. Retrieve the ID of the version retention policy for the bucket:

    root@rok-tools:~# export POLICY_ID=$(kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE} -o json \ > policy-list --bucket ${BUCKET_NAME?} | \ > jq -r -c ".[] | select(.action == \"version_retention\") | .policy_id")
  4. Display information about the retention policy. Verify that the scheduling rules and version retention rules match the default ones you have configured in the Configure Default Retention Policy for New Buckets guide:

    root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE?} policy-show ${POLICY_ID?} Policy Information Policy ID bea1e8a1-38fa-4786-a612-299f39ef060a Action version_retention Account kubeflow-user Bucket verify-retention Created At 2022-09-05T21:49:56.761403+00:00 Description Version Retention Policy Details Retention policy for versions in bucket `verify-retention' Paused False Current Run ID Next Run ID fbf4f09cbe114d7ca7f0f84ab1f6612f Next Run At 2022-09-09T00:00:00+00:00 Last Run ID bcac41be04094d12a2a005f27e29155c Last Run Status success Last Run Started At 2022-09-05T21:57:35.205674+00:00 Last Run Completed At 2022-09-05T21:57:35.560008+00:00 Scheduling Rule ID Interval Start Description 2e7c4c72-75da-442e-8f74-b8e5768701c5 1day 2022-09-05T00:00:00+00:00 Run every 1 day, starting at 2022-09-05T00:00:00+00:00 Version Retention Rule ID Strategy Params Description df216760-f2bd-4906-992f-8aacfc58d04b version_count 1,forever Retain the current version of each object forever e6602f83-d9c7-415b-887b-7872890e9c01 age all,1weeks Retain all versions of each object for 1 week 649c472a-f629-4d8a-b2df-845015269552 periodic 1per1weeks,2022-09-05T00:00:00+00:00,6months Retain one version of each object every 1 week (starting at 2022-09-05T00:00:00+00:00) for 6 months a1d912fb-6cac-4a8f-baa8-c1169f5cce0b periodic 1per3months,2022-09-01T00:00:00+00:00,5years Retain one version of each object every 3 months (starting at 2022-09-01T00:00:00+00:00) for 5 years

Summary

You have successfully updated the version retention policy of Rok API buckets in your EKF installation.

What’s Next

Check out the rest of the operations you can perform on your cluster.