Snapshot policies for Backup¶
Rok snapshot policies allow users to automatically create snapshots of their resources with a specified schedule. Additionally, version retention policies define how long each of these snapshots will be retained. The combination of the two enables users to restore their resources to any previous point in time, without the need to retain an unreasonably large amount of snapshots.
Snapshot policy for Jupyter notebooks¶
This section provides examples to create a snapshot policy for Jupyter notebooks, along with an accompanying version retention policy. The aim of the retention policy is to retain all recent snapshots, which minimizes the possibility of data loss and allows quick recovery from user errors, while at the same time retaining snapshots more sparsely the older they are, to limit their total number.
Rok UI¶
This section describes how to create a snapshot policy for Jupyter notebooks and configure the bucket’s version retention policy using the Rok UI.
Create a snapshot policy¶
Visit the Kubeflow dashboard, and go to Rok by selecting the
Snapshots
tab of the sidebar.(optional) Create a new bucket by clicking on the
Create bucket
button. Give it a name and click onCreate
. If you already have a bucket you want to use, you can skip this step.Click on the bucket, and select the policies tab.
Click the
+ Add policy
button next toSnapshot policies
.From the list of available drivers, select the JupyterLab driver.
Provide a name for the policy, and optionally a detailed description.
Set the
Backup policy
toggle toTrue
, to ensure the policy snapshots resources unconditionally, even if they have already been snapshotted.Define a schedule for your policy. This controls how often the policy runs, i.e., how often you create snapshots of your notebooks. In this example we will configure the policy to run every 15 minutes, effective immediately. This means that we will be creating 4 snapshots of each notebook every hour.
- Click the
+
icon under theSchedule
section. - Select the
Repeat
tab, to add a scheduling rule that runs periodically. - Enter
15
minute(s)
in the interval. - Optionally, you can specify the exact time the repeating runs will start,
or leave the
Starting now
default to have them start immediately.
- Click the
Define which notebooks the policy should snapshot. To achieve this, attach filters to your policy. Note that you are required to specify at least one filter that defines the namespace. In this example we will configure the policy to snapshot all notebooks in namespace
kubeflow-user
whose name starts withdev-
.- Click the
+
icon under theResources to Snapshot
section. - Configure the filter to limit the namespace to
kubeflow-user
by selectingNamespace
,equal
,kubeflow-user
in the three input boxes. - (optional) Click on
+
again to add a second filter, and enterJupyterLab
,starts_with
,dev-
in the three new input boxes. This will make the policy only snapshot notebooks whose names start withdev-
. If you omit this filter then all notebooks in the desired namespace will be snapshotted.
- Click the
Click the
Add
button to create the policy. The first task of the policy should now be visible in theTasks
tab of the bucket.
Configure the version retention policy¶
Visit the
Policies
tab again in the same bucket.Under the
Admin policies
section, click theedit
button on the version retention policy.Scroll down to the
Schedule
tab, and click on the+
button to add a new scheduling rule. Select theRepeat
tab to add a scheduling rule that runs periodically, and enter2
hour(s)
as its interval, to make the policy run every 2 hours.Scroll down to the
Retention
section which specifies which versions of each object the policy will retain, and delete the existing rule that retains all versions forever by clicking on the trashcan icon.Create five new retention rules by clicking on the
+
icon, and entering the following options:- Retain the current version forever.
- Retain all versions for
1
day(s)
- Retain one version every
1
hour(s)
for3
day(s)
- Retain one version every
12
hour(s)
for1
month(s)
- Retain one version every
1
day(s)
for3
month(s)
- Retain one version every
1
month(s)
for1
year(s)
Click the
Update
button to update the policy. You should be able to see the first task of the policy under theTasks
tab. Additionally, in theFiles
tab, the colored icon next to each file name now indicates its retention status, and hovering over it reveals information about the reasoning behind this decision.
Rok CLI¶
This section describes how to create a snapshot policy for Jupyter notebooks and configure the bucket’s version retention policy using the Rok CLI.
Note
For the purposes of this example we will be executing commands from
within a Jupyter notebook, where the Rok URL and Rok account, i.e.,
the --url
and --account
command line arguments, have already
been populated via the ROK_URL
and ROK_ACCOUNT
environment
variables.
Create a snapshot policy¶
(optional) Create a new bucket. If you already have a bucket you want to use, you can skip this step:
$ BUCKET="My bucket" $ rok bucket-create "${BUCKET?}"Create the snapshot policy for the bucket. The policy will run every 15 minutes and snapshot all notebooks in namespace
kubeflow-user
whose name starts withdev
:$ DESCRIPTION="Notebook snapshot policy" $ rok policy-create "${BUCKET?}" jupyter registration \ > --description "${DESCRIPTION?}" \ > --backup \ > --schedule now,15minutes \ > --filter params:namespace,equal,kubeflow-user \ > --filter params:lab,starts_with,dev-
Configure the version retention policy¶
Make sure you have
jq
installed:$ sudo apt-get install jqRetrieve the ID of the version retention policy of the bucket:
$ POLICY_ID=$(rok -o json policy-list --bucket "${BUCKET?}" | jq -r '.[] | select(.action=="version_retention") | .policy_id')Version retention policies have by default only a single retention rule to retain all versions of each object forever. Retrieve the IDs of all existing rules so that you can replace them with the desired ones:
$ DEL_RETENTION_ARGS=$(rok -o json policy-show ${POLICY_ID?} | jq -r '.action_params.rules[] | .version_retention_rule_id' | xargs -r -n1 echo --del-retention | xargs)Version retention policies have by default no scheduling rules. Retrieve the IDs of all scheduling rules so that you can replace them with the desired ones:
$ DEL_SCHEDULE_ARGS=$(rok -o json policy-show ${POLICY_ID?} | jq -r '.schedule[] | .scheduling_rule_id' | xargs -r -n1 echo --del-schedule | xargs)Remove the existing retention and scheduling rules and add new rules in order to
- Run every 2 hours.
- Retain the current of each object version forever.
- Retain all versions of each object for 1 day.
- Retain one version of each object every hour for 3 days.
- Retain one version of each object every 12 hours for 1 month.
- Retain one version of each object every day for 3 months.
- Retain one version of each object every month for 1 year.
$ rok policy-update ${POLICY_ID?} \ > ${DEL_RETENTION_ARGS?} \ > ${DEL_SCHEDULE_ARGS?} \ > --add-schedule now,2hours \ > --add-retention 1,forever \ > --add-retention all,1days \ > --add-retention 1per1hours,3days \ > --add-retention 1per12hours,1months \ > --add-retention 1per1days,3months \ > --add-retention 1per1months,1years