Configure Default Retention Policy for New Buckets¶
Version retention policies control the lifetime of snapshots inside a Rok API bucket. By default, the version retention policy of a new bucket will retain all versions forever. This guide will walk you through modifying this default and creating a retention policy with custom schedule and retention rules for every new bucket users create.
Note
The configuration you will perform in this guide will only affect the default settings with which version retention policies will be created for each new bucket. Users can still modify the version retention policy of their buckets after its creation via the Rok UI or Rok client.
Overview
What You’ll Need¶
- A configured management environment.
- Your clone of the Arrikto GitOps repository.
- An existing Kubeflow deployment.
Procedure¶
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:~# cd ~/ops/deploymentsModify how often the version retention policy will run. Edit the
rok/rok-cluster/overlays/deploy/patches/retention.yaml
file and set your desired time interval in the policy schedule. By default, the policy will have an empty schedule, meaning that it will never run.schedule: # <-- Replace the empty schedule with a scheduling rule - interval: 1 day # <-- specifying how often the policy will run rules: # Retain the latest version of each object forever - strategy: versionCount params: count: 1 retain: foreverNote
The schedule of a policy only affects how often the policy will run. It does not affect which versions the policy will delete once it runs. Policy intervals support the following units:
seconds
,minutes
,hours
,days
,weeks
,months
, andyears
.Modify which versions of each object the policy will retain. Edit the
rok/rok-cluster/overlays/deploy/patches/retention.yaml
file and replace the existing rule with a list of rules specifying how long you want versions to be retained.schedule: - interval: 1 day rules: # <-- Modify the items in this list # Retain the latest version of each object forever - strategy: versionCount params: count: 1 retain: forever # Retain all versions for 1 week - strategy: age params: retain: 1 week # Retain one version of each object per week for 6 months - strategy: periodic params: interval: 1 week retain: 6 months # Retain one version of each object per 3 months for 5 years - strategy: periodic params: interval: 3 months retain: 5 yearsNote
Each retention rule specifies a subset of the versions of each object that the policy will retain. Versions that are not retained by any rule will be deleted. The way each rule determines which versions to retain depends on its
strategy
andparams
.The supported strategies are the following:
versionCount
: This rule retains a specified number of versions of each object. This rule accepts the parameterscount
andretain
.age
: This rule retains all versions for a specified duration after their creation. This rule accepts only the parameterretain
.periodic
: This rule retains one version of each object per specified time interval. If multiple versions of the same object exist within the same time interval, the latest version is selected. This rule accepts the parametersinterval
,start
, andretain
.
The supported parameters are the following:
interval
: The time interval in which the rule will retain one version. For example, ifinterval
is set to2 hours
, the rule will retain one version in each two-hour window.start
: The starting timestamp of the retention interval, in ISO8601 format. Use this parameter to fine-tune when each interval of the policy will start. If omitted, the timestamp of the rule’s creation will be used.retain
: The duration for which versions will be retained. If omitted or set toforever
, the rule will retain the selected versions forever.count
: The number of versions the rule will retain. The latest versions available are always selected.
Commit your changes:
root@rok-tools:~/ops/deployments# git commit \ > -am "Configure default retention policy for new buckets"Apply the kustomization:
root@rok-tools:~/ops/deployments# rok-deploy \ > --apply rok/rok-cluster/overlays/deploy
Verify¶
Specify the namespace in which to create a Rok API bucket:
root@rok-tools:~# export NAMESPACE=<NAMESPACE>Replace
NAMESPACE
with the name of the namespace that you wish to create the bucket in, for example:root@rok-tools:~# export NAMESPACE=kubeflow-userSpecify the name of the bucket to create:
root@rok-tools:~# export BUCKET_NAME=<BUCKET_NAME>Replace
BUCKET_NAME
with the name of the bucket you wish to create, for example:root@rok-tools:~# export BUCKET_NAME=verify-retentionCreate the bucket:
root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE?} bucket-create ${BUCKET_NAME?} Bucket Information Name verify-retention Throw Type local Throw Task ID - Object Count 0 Bytes Used 0 Version Count 0 Version Bytes Used 0 Pending Object Count 0 Pending Bytes Used 0 Pending Version Count 0 Pending Version Bytes Used 0 Created At 2022-09-06T15:40:47.134839+00:00 Block Hash sha256 Block Size 4MiB Storage Policy Policy-0Troubleshooting
The bucket already exists
If the bucket you tried to create already exists, the above command will fail with the following error:
Error: Bucket 'verify-retention' already exists command terminated with exit code 1In this case, go back to step 2. and select a different bucket name.
Retrieve the ID of the version retention policy for the bucket:
root@rok-tools:~# export POLICY_ID=$(kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE} -o json \ > policy-list --bucket ${BUCKET_NAME?} | \ > jq -r -c ".[] | select(.action == \"version_retention\") | .policy_id")Display information about the retention policy. Verify that the scheduling rules in the output match the scheduling rules you specified in step 2 of the procedure, and the version retention rules match the rules you specified in step 3 of the procedure:
root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE?} policy-show ${POLICY_ID?} Policy Information Policy ID bea1e8a1-38fa-4786-a612-299f39ef060a Action version_retention Account kubeflow-user Bucket verify-retention Created At 2022-09-05T21:49:56.761403+00:00 Description Version Retention Policy Details Retention policy for versions in bucket `verify-retention' Paused False Current Run ID Next Run ID fbf4f09cbe114d7ca7f0f84ab1f6612f Next Run At 2022-09-09T00:00:00+00:00 Last Run ID bcac41be04094d12a2a005f27e29155c Last Run Status success Last Run Started At 2022-09-05T21:57:35.205674+00:00 Last Run Completed At 2022-09-05T21:57:35.560008+00:00 Scheduling Rule ID Interval Start Description 2e7c4c72-75da-442e-8f74-b8e5768701c5 1day 2022-09-05T00:00:00+00:00 Run every 1 day, starting at 2022-09-05T00:00:00+00:00 Version Retention Rule ID Strategy Params Description df216760-f2bd-4906-992f-8aacfc58d04b version_count 1,forever Retain the current version of each object forever e6602f83-d9c7-415b-887b-7872890e9c01 age all,1weeks Retain all versions of each object for 1 week 649c472a-f629-4d8a-b2df-845015269552 periodic 1per1weeks,2022-09-05T00:00:00+00:00,6months Retain one version of each object every 1 week (starting at 2022-09-05T00:00:00+00:00) for 6 months a1d912fb-6cac-4a8f-baa8-c1169f5cce0b periodic 1per3months,2022-09-01T00:00:00+00:00,5years Retain one version of each object every 3 months (starting at 2022-09-01T00:00:00+00:00) for 5 yearsDelete the bucket:
root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE?} bucket-delete ${BUCKET_NAME?}