Configure Default Snapshot Policy for Notebooks

This guide will walk you through configuring the default snapshot policy for Jupyter Notebooks in an Arrikto EKF installation. By default, the policy will snapshot all notebooks in all user namespaces once per day.

What You'll Need

Procedure

  1. Go to the skel-resources deploy overlay of your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments/kubeflow/manifests/common/skel-resources/overlays/deploy
    
  2. Include the notebook snapshot policy patch in the kustomization resources of the overlay:

    root@rok-tools:~/ops/deployments/kubeflow/manifests/common/skel-resources/overlays/deploy# kustomize \
    >    edit add patch patches/notebook-backup.yaml
    
  3. If you wish to modify the frequency with which the policy will run, edit the patches/notebook-backup.yaml file and set your desired time interval. By default, the policy will run once per day:

    schedule:
    - interval: 1 day  # <-- Set this line to your desired interval
    #filters:
    #- key: name
    #  operator: StartsWith
    #  value: prefix-
    #paused: false
    

    Note

    Rok policy intervals support the following units: seconds, minutes, hours, days, weeks, months, years.

  4. If you wish the policy to only snapshot a subset of the notebooks, follow these steps to configure a policy filter. If you wish to snapshot all notebooks, you may skip this step.

    1. Edit the patches/notebook-backup.yaml file and uncomment the filters:

      schedule:
      - interval: 1 day
      filters:                # <-- Uncomment this line
      - key: name             # <-- Uncomment this line
        operator: StartsWith  # <-- Uncomment this line
        value: prefix-        # <-- Uncomment this line
      #paused: false
      
    2. Set the operator of the filter to the comparison operator you wish to use to determine which notebooks to snapshot. For example, if the operator is equal to StartsWith, the policy will only snapshot notebooks whose name starts with the provided value:

      filters:
      - key: name
        operator: StartsWith  # <-- Update this line with your desired operator
        value: prefix-
      

      Note

      Rok policy filters support the following operators: Exists, Equal, NotEqual, Contains, NotContains, LessThan, LessThanOrEqual, GreaterThan, GreaterThanOrEqual, StartsWith, NotStartsWith, EndsWith, NotEndsWith.

    3. Set the value of the filter to the value you wish to compare against.

      filters:
      - key: name
        operator: StartsWith
        value: prefix-        # <-- Update this line with your value
      
  5. If you wish to temporarily disable the policy in all user namespaces, edit the patches/notebook-backup.yaml file, uncomment the paused field and set it to true:

    #filters:
    #- key: name
    #  operator: StartsWith
    #  value: prefix-
    paused: true   # <-- Uncomment this line and set it to true
    

    Note

    Paused policies do not schedule any new runs. You can restore the value of this field at any point to resume the policies.

  6. Return to the base directory of your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~/ops/deployments/kubeflow/manifests/common/skel-resources/overlays/deploy# cd ~/ops/deployments
    
  7. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit \
    >    -am "Configure default snapshot policy for notebooks"
    
  8. Apply the kustomization:

    root@rok-tools:~/ops/deployments# rok-deploy \
    >    --apply kubeflow/manifests/common/skel-resources/overlays/deploy
    

Verify

  1. Retrieve the policy in the kubeflow-skel namespace:

    root@rok-tools:~# kubectl get -n kubeflow-skel roksnapshotpolicy -o wide
    NAME              KIND       BUCKET         POLICY ID                              PAUSED   AGE
    notebook-backup   Notebook   auto-backups   010660d8-6e7d-4afc-ac51-ad165876b7a4            10m
    
  2. Verify that the policy KIND is equal to Notebook.

  3. Verify that the POLICY ID is not empty.

    Note

    Repeat the following steps for every namespace you wish to verify.

  4. Specify the namespace you wish to verify:

    root@rok-tools:~# export NAMESPACE=<NAMESPACE>
    

    Replace NAMESPACE with the name of the namespace that you wish to verify, for example:

    root@rok-tools:~# export NAMESPACE=kubeflow-user
    
  5. Retrieve the policy:

    root@rok-tools:~# kubectl get -n ${NAMESPACE?} roksnapshotpolicy -o wide
    NAME              KIND       BUCKET         POLICY ID                              PAUSED   AGE
    notebook-backup   Notebook   auto-backups   43f35af5-5827-4731-b020-6cc5bac38a30             5m
    
  6. Verify that the policy KIND is equal to Notebook.

  7. Verify that the POLICY ID is not empty.

Summary

You have successfully created a snapshot policy in all user namespaces.

What's Next

Check out the rest of the operations you can perform on your Kubeflow deployment.