Configure Default Retention Policy for New Buckets

Version retention policies control the lifetime of snapshots inside a Rok API bucket. By default, the version retention policy of a new bucket will retain all versions forever. This guide will walk you through modifying this default and creating a retention policy with custom schedule and retention rules for every new bucket users create.

Note

The configuration you will perform in this guide will only affect the default settings with which version retention policies will be created for each new bucket. Users can still modify the version retention policy of their buckets after its creation via the Rok UI or Rok client.

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Modify how often the version retention policy will run. Edit the rok/rok-cluster/overlays/deploy/patches/retention.yaml file and set your desired time interval in the policy schedule. By default, the policy will have an empty schedule, meaning that it will never run.

    schedule: # <-- Replace the empty schedule with a scheduling rule - interval: 1 day # <-- specifying how often the policy will run rules: # Retain the latest version of each object forever - strategy: versionCount params: count: 1 retain: forever

    Note

    The schedule of a policy only affects how often the policy will run. It does not affect which versions the policy will delete once it runs. Policy intervals support the following units: seconds, minutes, hours, days, weeks, months, and years.

  3. Modify which versions of each object the policy will retain. Edit the rok/rok-cluster/overlays/deploy/patches/retention.yaml file and replace the existing rule with a list of rules specifying how long you want versions to be retained.

    schedule: - interval: 1 day rules: # <-- Modify the items in this list # Retain the latest version of each object forever - strategy: versionCount params: count: 1 retain: forever # Retain all versions for 1 week - strategy: age params: retain: 1 week # Retain one version of each object per week for 6 months - strategy: periodic params: interval: 1 week retain: 6 months # Retain one version of each object per 3 months for 5 years - strategy: periodic params: interval: 3 months retain: 5 years

    Note

    Each retention rule specifies a subset of the versions of each object that the policy will retain. Versions that are not retained by any rule will be deleted. The way each rule determines which versions to retain depends on its strategy and params.

    The supported strategies are the following:

    • versionCount: This rule retains a specified number of versions of each object. This rule accepts the parameters count and retain.
    • age: This rule retains all versions for a specified duration after their creation. This rule accepts only the parameter retain.
    • periodic: This rule retains one version of each object per specified time interval. If multiple versions of the same object exist within the same time interval, the latest version is selected. This rule accepts the parameters interval, start, and retain.

    The supported parameters are the following:

    • interval: The time interval in which the rule will retain one version. For example, if interval is set to 2 hours, the rule will retain one version in each two-hour window.
    • start: The starting timestamp of the retention interval, in ISO8601 format. Use this parameter to fine-tune when each interval of the policy will start. If omitted, the timestamp of the rule’s creation will be used.
    • retain: The duration for which versions will be retained. If omitted or set to forever, the rule will retain the selected versions forever.
    • count: The number of versions the rule will retain. The latest versions available are always selected.
  4. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit \ > -am "Configure default retention policy for new buckets"
  5. Apply the kustomization:

    root@rok-tools:~/ops/deployments# rok-deploy \ > --apply rok/rok-cluster/overlays/deploy

Verify

  1. Specify the namespace in which to create a Rok API bucket:

    root@rok-tools:~# export NAMESPACE=<NAMESPACE>

    Replace NAMESPACE with the name of the namespace that you wish to create the bucket in, for example:

    root@rok-tools:~# export NAMESPACE=kubeflow-user
  2. Specify the name of the bucket to create:

    root@rok-tools:~# export BUCKET_NAME=<BUCKET_NAME>

    Replace BUCKET_NAME with the name of the bucket you wish to create, for example:

    root@rok-tools:~# export BUCKET_NAME=verify-retention
  3. Create the bucket:

    root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE?} bucket-create ${BUCKET_NAME?} Bucket Information Name verify-retention Throw Type local Throw Task ID - Object Count 0 Bytes Used 0 Version Count 0 Version Bytes Used 0 Pending Object Count 0 Pending Bytes Used 0 Pending Version Count 0 Pending Version Bytes Used 0 Created At 2022-09-06T15:40:47.134839+00:00 Block Hash sha256 Block Size 4MiB Storage Policy Policy-0

    Troubleshooting

    The bucket already exists

    If the bucket you tried to create already exists, the above command will fail with the following error:

    Error: Bucket 'verify-retention' already exists command terminated with exit code 1

    In this case, go back to step 2. and select a different bucket name.

  4. Retrieve the ID of the version retention policy for the bucket:

    root@rok-tools:~# export POLICY_ID=$(kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE} -o json \ > policy-list --bucket ${BUCKET_NAME?} | \ > jq -r -c ".[] | select(.action == \"version_retention\") | .policy_id")
  5. Display information about the retention policy. Verify that the scheduling rules in the output match the scheduling rules you specified in step 2 of the procedure, and the version retention rules match the rules you specified in step 3 of the procedure:

    root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE?} policy-show ${POLICY_ID?} Policy Information Policy ID bea1e8a1-38fa-4786-a612-299f39ef060a Action version_retention Account kubeflow-user Bucket verify-retention Created At 2022-09-05T21:49:56.761403+00:00 Description Version Retention Policy Details Retention policy for versions in bucket `verify-retention' Paused False Current Run ID Next Run ID fbf4f09cbe114d7ca7f0f84ab1f6612f Next Run At 2022-09-09T00:00:00+00:00 Last Run ID bcac41be04094d12a2a005f27e29155c Last Run Status success Last Run Started At 2022-09-05T21:57:35.205674+00:00 Last Run Completed At 2022-09-05T21:57:35.560008+00:00 Scheduling Rule ID Interval Start Description 2e7c4c72-75da-442e-8f74-b8e5768701c5 1day 2022-09-05T00:00:00+00:00 Run every 1 day, starting at 2022-09-05T00:00:00+00:00 Version Retention Rule ID Strategy Params Description df216760-f2bd-4906-992f-8aacfc58d04b version_count 1,forever Retain the current version of each object forever e6602f83-d9c7-415b-887b-7872890e9c01 age all,1weeks Retain all versions of each object for 1 week 649c472a-f629-4d8a-b2df-845015269552 periodic 1per1weeks,2022-09-05T00:00:00+00:00,6months Retain one version of each object every 1 week (starting at 2022-09-05T00:00:00+00:00) for 6 months a1d912fb-6cac-4a8f-baa8-c1169f5cce0b periodic 1per3months,2022-09-01T00:00:00+00:00,5years Retain one version of each object every 3 months (starting at 2022-09-01T00:00:00+00:00) for 5 years
  6. Delete the bucket:

    root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE?} bucket-delete ${BUCKET_NAME?}

Summary

You have successfully modified the default version retention policy for new Rok API buckets.

What’s Next

Check out the rest of the operations you can perform on your cluster.