Command execution

The rok_common.cmdutils module provides some extra features on top of Python’s subprocess module that make command execution more flexible and platform-agnostic. In case you need to use Python to invoke and orchestrate CLI tools, we recommend using this library over the standard Python API for command execution.

Note

rok_common is available in any notebook (or related) image provided by Arrikto in your Kubeflow deployment. To test the examples in the following sections, create a new Notebook from the Kubeflow UI using the available jupyter-kale image.

Examples

Below follow some sample usages of the rok_common.cmdutils library:

Import the module

>>> from rok_common import cmdutils

Run a command and print to the console

>>> c = cmdutils.run("which echo") /bin/echo >>> c <ExtCommand [MjY3bxTC7vY] `which echo', status=FINISHED (ret: 0), PID=18236, shell=False>

In case of an error, run() raises an exception:

>>> cmdutils.run("ls /does/not/compute") ls: cannot access '/does/not/compute': No such file or directory [vK8UUGJfmDE] Command failed with ret: 2. Stderr was not captured. Traceback (most recent call last): ... rok_common.cmdutils.CommandExecutionError: Command `<ExtCommand [vK8UUGJfmDE] `ls /does/not/compute', status=FINISHED (ret: 2), PID=20501, shell=False>' failed

Run a command and capture its output

>>> out = cmdutils.yld("which echo") >>> out '/bin/echo'

In case of an error, yld() raises an exception and reports the stderr as well:

>>> out = cmdutils.yld("ls /does/not/compute") [xj8HgonPWQk] Command failed with ret: 2. Error log: ls: cannot access '/does/not/compute': No such file or directory Traceback (most recent call last): ... rok_common.cmdutils.CommandExecutionError: Command `<ExtCommand [xj8HgonPWQk] `ls /does/not/compute', status=FINISHED (ret: 2), PID=20243, shell=False>' failed. Error log: ls: cannot access '/does/not/compute': No such file or directory\n

Run a command and capture its output, while printing to the console

>>> import sys >>> c = cmdutils.run("which echo", stdout=[cmdutils.STORE, sys.stdout], ... stderr=[cmdutils.STORE, sys.stderr]) /bin/echo >>> c.out '/bin/echo\n'

In case of an error, run() raises an exception, and both captures and reports the stderr in the error message as well:

>>> cmdutils.run("ls /does/not/compute", ... stdout=[cmdutils.STORE, sys.stdout], ... stderr=[cmdutils.STORE, sys.stderr]) ls: cannot access '/does/not/compute': No such file or directory [xj8HgonPWQk] Command failed with ret: 2. Error log: ls: cannot access '/does/not/compute': No such file or directory Traceback (most recent call last): ... rok_common.cmdutils.CommandExecutionError: Command `<ExtCommand [xj8HgonPWQk] `ls /does/not/compute', status=FINISHED (ret: 2), PID=20243, shell=False>' failed. Error log: ls: cannot access '/does/not/compute': No such file or directory\n

Run a command and write to its stdin

>>> cmdutils.yld("sha256sum", input="test") '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 -'

Run a command asynchronously

>>> c = cmdutils.run("sleep 2", wait=False) >>> c.wait()

Run a command with a timeout

>>> cmdutils.yld("sleep infinity", timeout=1) [h3EPxWt0B1A] Command did not terminate after 1.00 seconds Traceback (most recent call last): ... rok_common.cmdutils.CommandTimeoutError: Command `<ExtCommand [h3EPxWt0B1A] `sleep infinity', status=RUNNING, PID=22124, shell=False>' timed out after 1.00s

Run a command and preserve whitespace

>>> cmdutils.yld(["ls", "/tmp/file with spaces"]) '/tmp/file with spaces'

You should prefer the list form when the command contains user-controlled variables, or when whitespace should be treated literally. The same example in a string form fails:

>>> cmdutils.yld("ls /tmp/file with spaces") [vRhtfmASMxw] Command failed with ret: 2. Error log: ls: cannot access '/tmp/file': No such file or directory ls: cannot access 'with': No such file or directory ls: cannot access 'spaces': No such file or directory Traceback (most recent call last): ... rok_common.cmdutils.CommandExecutionError: Command `<ExtCommand [vRhtfmASMxw] `ls /tmp/file with spaces', status=FINISHED (ret: 2), PID=14434, shell=False>' failed. Error log: ls: cannot access '/tmp/file': No such file or directory\nls:cannot access 'with': No such file or directory\nls: cannot access 'spaces': No such file or directory\n

Note that you can make it work with a bit of extra care, but in general you should avoid doing so:

>>> cmdutils.yld("ls '/tmp/file with spaces'") '/tmp/file with spaces'

Enable logging

We propose the INFO logging level for the cmdutils module:

>>> import logging >>> logging.basicConfig(level=logging.INFO) >>> cmdutils.yld("which echo") INFO:rok_common.cmdutils:[8qmHl1CZfwQ] Started command with PID 22775: which echo INFO:rok_common.cmdutils:[8qmHl1CZfwQ] Command exited with ret: 0 '/bin/echo'

In DEBUG, the logs are more verbose:

>>> import logging >>> logging.basicConfig(level=logging.DEBUG) >>> cmdutils.yld("which echo") DEBUG:rok_common.cmdutils:[uheXgEB-8BM] Starting command: which echo INFO:rok_common.cmdutils:[uheXgEB-8BM] Started command with PID 23086: which echo DEBUG:rok_common.cmdutils:[uheXgEB-8BM] Waiting command INFO:rok_common.cmdutils:[uheXgEB-8BM] Command exited with ret: 0 DEBUG:rok_common.cmdutils:[uheXgEB-8BM] Cleaning command DEBUG:rok_common.cmdutils:[uheXgEB-8BM] Command execution finished '/bin/echo'

Create a generic command that accepts parameters

>>> class Find(ExtCommand): ... ... command = "find %(path)s -name %(name)s" ... ... def __init__(self, path, name): ... self.path = path ... self.name = name >>> extcom = Find(path="/usr", name="python*") >>> extcom.start(wait=False) >>> print extcom.status RUNNING >>> print extcom.out /usr/bin/python /usr/bin/python2 >>> extcom.wait() >>> print extcom.out /usr/bin/python /usr/bin/python2 /usr/bin/python2.7 /usr/lib/python2.7 >>> print extcom.status FINISHED >>> print extcom.returncode 0