Switch Release Channel

This section describes how to switch from release-2.0 to develop release channel for EKF.

What You’ll Need

  • An upgraded management environment.
  • Your local clone of the Arrikto GitOps repository with manifests for EKF 2.0.1 or later.

Check Your Environment

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Check your current release channel:

    root@rok-tools:~/ops/deployments# git rev-parse --abbrev-ref HEAD release-2.0
  3. Ensure that you are currently on the release-2.0 branch:

    root@rok-tools:~/ops/deployments# git rev-parse --abbrev-ref HEAD release-2.0
  4. Verify that your GitOps repository is clean:

    root@rok-tools:~/ops/deployments# git status On branch release-2.0 nothing to commit, working tree clean
  5. Ensure that you are currently following the release-2.0 release channel:

    root@rok-tools:~/ops/deployments# git rev-parse --abbrev-ref --symbolic-full-name @{u} origin/release-2.0

Procedure

Important

Mirror the GitOps repository to a private remote to be able to recover it in case of failure.

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Run keychain to start the SSH agent and use your SSH identity:

    root@rok-tools:~/ops/deployments# eval "$(keychain --eval id_rsa)" * keychain 2.8.2 ~ http://www.funtoo.org * Starting ssh-agent... * Adding 1 ssh key(s): /root/.ssh/id_rsa * ssh-add: Identities added: /root/.ssh/id_rsa

    Troubleshooting

    can’t find id_rsa

    If the above command shows the following warning message:

    Warning: can't find id_rsa; skipping

    it means that keychain could not find the file ~/.ssh/id_rsa, which normally contains your private SSH key.

    To proceed, follow the Configure Git guide to configure your private SSH key.

    Cannot find public key for id_rsa

    If the above command shows the following warning message:

    Warning: Cannot find public key for id_rsa

    it means that keychain clould not find the file ~/.ssh/id_rsa.pub, which normally contains your public SSH key.

    To proceed, generate the public SSH key based on your private key:

    root@rok-tools:~# ssh-keygen -yf /root/.ssh/id_rsa > /root/.ssh/id_rsa.pub
  3. Fetch latest upstream changes:

    root@rok-tools:~/ops/deployments# git fetch --all -p Fetching origin

    Troubleshooting

    Permission denied

    If the above command fails with the following error message:

    Permission denied (publickey)

    ensure that you have run keychain as specified in the Procedure or the Fast-Forward section of this guide.

    If the error persists, it means that you do not have access to the Arrikto GitOps repository. To proceed, follow the Configure Git guide to configure your SSH identity.

  4. Ensure that the release-2.0 release channel you are currently following contains only Arrikto commits:

    root@rok-tools:~/ops/deployments# [ $(git rev-list origin/release-2.0 --invert-grep --author @arrikto.com --count) == 0 ] \ > && echo "OK" || echo "FAIL" OK

    Troubleshooting

    The command outputs FAIL

    If the above command outputs FAIL then the upstream branch release-2.0 you are following contains custom (non-Arrikto) commits which is not expected. This can happen if you, for example, push custom (non-Arrikto) commits to a mirror of an Arrikto-managed branch.

    Caution

    Before proceeding please contact the Arrikto Tech Team to ensure that your GitOps repository is in a good state for the upgrade.

  5. Ensure that the develop release channel you are about to switch to contains only Arrikto commits:

    root@rok-tools:~/ops/deployments# [ $(git rev-list origin/develop --invert-grep --author @arrikto.com --count) == 0 ] \ > && echo "OK" || echo "FAIL" OK

    Troubleshooting

    The command outputs FAIL

    If the above command outputs FAIL then the upstream branch develop you are about to switch to contains custom (non-Arrikto) commits which is not expected. This can happen if you, for example, push custom (non-Arrikto) commits to a mirror of an Arrikto-managed branch.

    Caution

    Before proceeding please contact the Arrikto Tech Team to ensure that your GitOps repository is in a good state for the upgrade.

  6. Prepare your branch for the upcoming rebase, with the rok-kf-rebase tool:

    root@rok-tools:~/ops/deployments# rok-kf-rebase
  7. After rok-kf-rebase completes, verify that the intermediate rebase branch was created:

    root@rok-tools:~/ops/deployments# git show --shortstat develop-base commit 7ac4720df93325399aea44519bf69b55fec83e03 (develop-base) Author: Rok Kubeflow Rebase <no-reply@arrikto.com> Date: Wed May 12 10:49:52 2021 +0000 Renaming commit

    Note

    This commit might be empty if no renames are required.

  8. Switch to the develop release channel and rebase your work on top of it:

    root@rok-tools:~/ops/deployments# git \ > rebase -Xtheirs --onto origin/develop develop-base

    Troubleshooting

    CONFLICT (modify/delete)

    Rebasing your work may cause conflicts when you have modified a file that has been removed from the latest version of Arrikto manifests. In such a case the rebase process will fail with:

    CONFLICT (modify/delete): path/to/file deleted in origin/develop and modified in HEAD~X. Version HEAD~X of path/to/file left in tree.
    1. Delete those files:

      root@rok-tools:~/ops/deployments# git status \ > --porcelain \ > | awk '{if ($1=="DU") print $2}' \ > | xargs git rm
    2. Continue the rebase process:

      root@rok-tools:~/ops/deployments# git rebase --continue
  9. Configure your local branch to track the develop release channel:

    root@rok-tools:~/ops/deployments# git branch --set-upstream-to=origin/develop
  10. Delete the intermediate rebase branch:

    root@rok-tools:~/ops/deployments# git branch -D develop-base

Important

rok-kf-rebase has reverted custom changes in the JWA ConfigMap patch. If you had added custom JWA images you have to readd them by following the next steps:

  1. See the reverted changes:

    root@rok-tools:~/ops/deployments# git diff release-2.0 -- \ > kubeflow/manifests/apps/jupyter/jupyter-web-app/upstream/overlays/deploy/patches/config-map.yaml
  2. Follow the Extend List of JWA Default Images guide to readd the desired changes.

Important

rok-kf-rebase has reverted changes in the cluster-autoscaler Kustomization.

  1. See the reverted changes:

    root@rok-tools:~/ops/deployments# git diff release-2.0 -- \ > rok/cluster-autoscaler/overlays/deploy
  2. Follow the Deploy Cluster Autoscaler guide to reconfigure it.

Verify

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Ensure that you are currently on the develop branch:

    root@rok-tools:~/ops/deployments# git rev-parse --abbrev-ref HEAD develop
  3. Get the current status of your GitOps repository and verify that your local repository is clean:

    root@rok-tools:~/ops/deployments# git status On branch develop nothing to commit, working tree clean
  4. Ensure that you are currently following the develop release channel:

    root@rok-tools:~/ops/deployments# git rev-parse --abbrev-ref --symbolic-full-name @{u} origin/develop

Summary

You have successfully switched to the latest release channel and set up your GitOps repository to track it.

What’s Next

The next step is to update your manifests.