Enable or Disable Kiwi Scheduler

This guide will instruct you on how to toggle Kiwi’s exclusive access mechanism on and off.

The exclusive access mechanism, implemented by the Kiwi Scheduler, exists in order to prevent thrashing scenarios when the sum of the applications’ working set sizes (GPU memory) exceeds the physical GPU memory capacity.

Important

By default, the Kiwi Scheduler is enabled. Disabling it without ensuring that the working sets (GPU memory) of collocated applications fit in GPU memory can cause thrashing and severe performance degradation.

Important

Changes to the configuration of a Kiwi Scheduler instance only affect that particular instance and are not persisted across Pod restarts.

In order to make the changes persistent and also have them apply to all Kiwi Scheduler instances you must redeploy Kiwi with the desirable configuration.

What You’ll Need

Procedure

  1. Find the node for which you wish to change the Kiwi Scheduler’s status.

    1. Specify the name of the application Pod you are interested in changing the Kiwi Scheduler’s status for:

      root@rok-tools:~# KIWI_POD_NAME=<POD_NAME>

      Replace <POD_NAME> with the name of the Pod you want to configure, for example:

      root@rok-tools:~# KIWI_POD_NAME=kiwi-pod
    2. Specify the namespace of the application Pod:

      root@rok-tools:~# KIWI_POD_NAMESPACE=<POD_NAMESPACE>

      Replace <POD_NAMESPACE> with the namespace of the Pod you want to configure, for example:

      root@rok-tools:~# KIWI_POD_NAMESPACE=kiwi-pod-namespace
    3. Find the node where the application Pod is running on:

      root@rok-tools:~# KIWI_NODENAME=$(kubectl get pod ${KIWI_POD_NAME?} \ > -n ${KIWI_POD_NAMESPACE?} -o json \ > | jq -r '.spec.nodeName') > && echo ${KIWI_NODENAME?} ip-192-168-109-143.eu-central-1.compute.internal
  2. Get the Kiwi Scheduler’s Pod name for the specified node:

    root@rok-tools:~# KIWI_SCHEDULER_POD_NAME=$(kubectl get pod \ > -n kiwi-system -l name=kiwi-scheduler -o json \ > | jq -r '.items[] | select(.spec.nodeName == "'$KIWI_NODENAME'") | .metadata.name') > && echo ${KIWI_SCHEDULER_POD_NAME?} kiwi-scheduler-4pk55
  3. Toggle the scheduler on or off. Choose one of the following options depending on your preference:

    root@rok-tools:~# kubectl exec -it ${KIWI_SCHEDULER_POD_NAME?} \ > -n kiwi-system -- kiwictl -S on [INFO] Successfully turned on the Kiwi scheduler.
    root@rok-tools:~# kubectl exec -it ${KIWI_SCHEDULER_POD_NAME?} \ > -n kiwi-system -- kiwictl -S off [INFO] Successfully turned off the Kiwi scheduler.

Verify

  1. Inspect the Kiwi Scheduler logs and verify that the scheduler’s status is equal to the one you set previously:

    root@rok-tools:~# kubectl logs ${KIWI_SCHEDULER_POD_NAME?} -n kiwi-system [INFO] Received SCHED_{ON,OFF} from <UNREGISTERED> [FD=5] [INFO] Scheduler status changed to {ON,OFF}, broadcasting it ...

Summary

You have successfully enabled or disabled the Kiwi scheduling mechanism.

What’s Next

Check out the rest of Kiwi operations you can perform.